Sunday, April 26, 2020

Overview of Real-Time CDP

Companies are continuously innovating and coming up with new tools and technologies to enhance the experience of its customers. Optimizing and personalizing the experience of customers at every touchpoint needs to be the holy grail for all companies. I gave an overview of a Data Management Platform a few years ago and have written a few article about Adobe Audience Manager since then. 

In this post, I'll write about Real-Time Customer Data Platform which is the activation engine and a service built on Adobe Experience Platform and how it solves the ever evolving challenge of a consistent and personalized user experience. As you may be aware, Adobe Experience Platform is built with an API-first approach which makes integrating with other platforms easier and it also allows us to replicate what we can do in the UI via APIs as well.


It's always helpful to start simple so I'll give a brief overview of what a Customer Data Platform is. There's a lot of documentation which compares a CDP to a DMP so I won't go into that in detail but will just focus on what a CDP is.


According to Wikipedia, "A customer data platform is a type of packaged software which creates a persistent, unified customer database that is accessible to other systems. Data is pulled from multiple sources, cleaned and combined to create a single customer profile. This structured data is then made available to other marketing systems".


I define it as a system which consolidates known and anonymous customer information from multiple data sources to serve 1:1 real-time personalized experiences across multiple touch points and marketing channels in a privacy aware manner. Below is a visual overview of a CDP (taken from a 3rd party site). The diagram looks very similar to a DMP but a CDP differs with a DMP in the following ways:

  • The activation of both known (PII) and anonymous user data is possible in a CDP
  • Profiles have a much longer data retention period in a CDP
  • A complete customer profile (historical, demographic) is maintained in a CDP



Core Elements of Real-Time CDP


Below are some core components of the Real-Time CDP and how it relates to Adobe Experience Platform.
  • XDM Schema & Datasets: The XDM Schema describes the logical model of all data coming into Adobe Experience Platform and contains information about the identities used to build the id graph. Essentially, any data that will be activated by Real-Time CDP has to be mapped to the XDM schema format. Datasets are tied to a schema and collect data either sent either via streaming or batch format from various sources (E.g. Audience Manager, Analytics, 3rd party sources) into AEP. 
  • Real-time Customer Profile: The Real-time Customer Profile is a combined view of all customer identities collected across multiple channels. I recently wrote about the device graph which creates a holistic view of the customer across multiple touch points. Real-Time CDP taps into the device/id graph to create a unified view of both the known and unknown customer profile on the fly depending on the use case.

  • Real-Time Web Data Streaming: The introduction of the AEP Web SDK has changed the dynamics of how Adobe data collection will happen moving forward. This essentially allows clients to send data directly to AEP in real-time as well as to Analytics, Audience Manager and Target from our website or mobile app. A lot of new content is being created on this which I'll share soon.

  • Real-Time Activation of Profiles: Real-Time CDP allows us to send segments and PII profiles to external destinations quickly keeping in mind the rapid pace at how customers can interact with brands across multiple channels. 
  • Built With Privacy By Design: All Adobe products including Real-Time CDP is built keeping privacy in mind. In case of the AEP UI, customers can use Data Usage Labeling and Enforcement (DULE) labels to mark mark identities or restrict how data will be activated based on GDPR and other privacy laws. The official Adobe documentation provides more details on this.

A lot more information can be found in the official Adobe documentation in addition to what I've covered.


Adobe Solution Integration with Real-Time CDP


Visuals always help so I created a quick high-level data flow diagram outlining how data flows into and outside of AEP/Real-Time CDP from various Adobe solutions. Only the arrows coming into and flowing out of AEP/Real-Time CDP are colored and these connections can be enabled OOTB. The diagram should be self-explanatory but web data flows into AEP/Real-Time CDP via the AEP Web SDK/Adobe Analytics and 2nd/3rd party/other data flows into Real-Time CDP from Audience Manager. Data from Real-Time CDP can be shared with Adobe Campaign to personalize emails.



Real-Time CDP UI Overview


I'll now provide a high level overview of the Real-Time CDP UI but please be aware that these are likely to change as product is continuously making changes to optimize the user experience and add more enhancements.


Profiles

A profile is a collection of an individual or entity which can be used to uniquely identify it. In this screenshot, the profile is tied to the ECID identity but there can be additional identities for this profile such as email address, phone number to name a few. This is embedded within the Adobe Experience Platform UI.



Segments

The segmentation UI for Real-Time CDP is the same interface which AEP uses and it looks very similar to the Segment Builder UI embedded in Adobe Analytics. Segments allow us to dissect very large amount of profiles into smaller and much more manageable chunks. 



Identities

The identity page contains identity namespaces which contain unique information about a person. As shown below, an identity namespace can be Email, address Phone number, ECID etc. This is also embedded within the Adobe Experience Platform UI.


Sources

The Sources->Catalog tab allows us to bring data into AEP/Real-Time CDP from various sources such as Adobe applications, CRM, Marketing automation to name a few. Data from these can be combined with each other and sent to various activation platforms. Again, this is also embedded within the Adobe Experience Platform UI.



Destinations

The Destinations->Catalog tab allows us to activate the audiences and share them with ESPs, marketing automation and advertising platforms. This is the core feature of Real-Time CDP which you won't see present in the regular AEP UI. There are two ways to send data via destinations:
  • Profile Export which allows us to export PII data to ESPs and other platforms. 
  • Segment Export (anonymous) which allows us to export profiles which qualified for a segment to Demand Side Platforms and is similar to how Audience Manager exports data to DSPs. 
This UI as shown below shows us the standard connections but also lets us connect to Launch which allows us to forward raw signals captured in Launch using extensions to any 3rd party partner.



The following is a visual (representational) system view which shows how many profiles are connected to sources and are sent to a destination.



Finally, I want to emphasize that I've not heard anything about Adobe Audience Manager being impacted by Real-Time CDP as we s
till need AAM for 2nd and 3rd party data activation. Also, Audience Manager has over 100 destinations whereas Real-Time CDP only has a few (currently) with connections to CRM, cloud providers and email systems but the list will only grow. I don't know about the overall product vision around the total numbers destinations and AAM in general but I'm sure our product team will clarify that.

So, that was it! I'm sure I've left out other finer details about Real-Time CDP but this tool is continuing to evolve and will only get better as its adoption increases.

Monday, March 9, 2020

Overview of Adobe Device Graph

We are surrounded by all kinds of (regular and smart) devices. It's either a Laptop, Tablet, Smart Phone, Smart Watch, Smart TV, Smart Vacuum, Smart Plug, Nest Thermostat, Wifi Camera or Chromecast to name a few. However, not all of these devices are truly connected due to many factors with one being that not all devices may belong to one single company in a household but the most obvious one is the lack of a true user "Identity". The other issue we face is that it's really difficult to track users and attribute an action across all these devices consistently. Google does this well with Google Home where it's able to sync my Nest thermostat with Chromecast using my Gmail ID. The other company is obviously Adobe ;)

By 2030, Americans will own as many as 15 connected devices. While we're not fully there yet, the introduction of new devices every year might expedite that trend. In 2016, Adobe's coined the phrase "Devices don't buy products, people do" which is still very relevant in 2020 and will be in the future. So how does Adobe establish the link between multiple devices to one person? In this post, I'll provide an overview of the various ways by which Adobe is able to link multiple devices to a person or household.


Concept of a Device Graph


According to this article, "A device graph, also known as “identity management,” is a map that links an individual to all the devices they use, which could be a person’s computer at work, laptop at home, tablet and smartphone". I'd like to add other smart devices to this list but the most common use cases for cross device analytics we see today are still around mobile, desktop and tablet but there's continuous innovation happening to bring different types of devices into the mix. 


Probabilistic and Deterministic Linking


The Adobe device graph comprises of multiple devices linked together via two methods namely deterministic and probabilistic. This Adobe article explains this concept really well but at a high level, probabilistic device link allows us to predict a person's identity (John Doe in our case) based on IP address, operating system etc. and deterministic device link allows us to identify a person based on their encrypted user ID captured on the website sent over to the Experience Cloud ID Service across devices. Please note that this matching and attribution happens 


The visual below is my attempt to explain the device graph at a high level where John Doe visits a website or mobile app using his iPhone, iPad and Mac. There are prettier visuals available in the Adobe documentation but I just wanted to create something simple to convey the concept.






Types of Device Graphs


There are primarily two types of device graphs which Adobe supports and these are ways by which you can create a true identity of your customers across multiple devices and sites. There's also an external graph option as well which allows companies to leverage 3rd party device graph data which is explained in detail here but it's not in scope for this post.


Device Co-Op


The Device Co-Op (available in US and Canada) allows companies to participate in a device graph which allows them to identify their "linked" customer devices (at a person and household level) across a magnitude of channels and websites in near real-time. The last time I heard about the scale of the Co-op, there were 100+ companies, about 300 Million users and 2 billion devices part of the Co-op device graph but this number is obviously higher now. 



The concept of the Co-op can be better understood based on the visual (above) taken from the Adobe blog

  • A customer John Doe visits the travel website and authenticates on both his mobile device and laptop. 
  • He then goes to a retail website but doesn't authenticate so the retailer doesn't know who this customer is.
  • Device co-op enables the retail website to "link" the customer's devices assuming both websites (companies) participate in the Device Co-op and identifies the anonymous user as John Doe. 
  • This in turn allows both companies to personalize the experience for this customer on both devices and websites. As you can see, the Device Co-op makes use of both the probabilistic and deterministic linking methods.

In contrast, if these companies didn't participate in the Co-op, then the retail company wouldn't be able to know what all devices did its customers use before purchasing something and treat it as 2 unique visitors instead of 1.

Below are some common use cases of the Device Co-op:

  • If you want to access to a large pool of users for prospecting related use cases.
  • If you want to perform frequency capping across devices.
  • Some other use cases are covered here.

Here's a link to a document which covers the membership and eligibility criteria for companies to participate in the Device Co-op. One other requirement of the Device Co-op is for the company to make changes to its privacy policy so it takes into account all the necessary privacy requirements and it does not collect any kind of PII or behavioral data. You can view your all your devices linked to the Co-op here and can unlink your devices from it anytime.

Private Device Graph


The Private Graph is another way for a company to link their customer devices in a device graph but visible and accessible only for their own organization and not any other company. 

Using the previous example, if we just look at the travel website alone we can treat it as a participant of the private graph as the data will simply be available within the context of that company. Private Device graph primarily makes use of the deterministic link as it works best if encrypted user ids are captured. Probabilistic matching will also be available to Private Graph sometime this year. 

Below are some use cases of the Private Device Graph:

  • If you want to reconcile user identities across multiple devices into one.
  • If you want to perform frequency capping across devices thereby optimizing ad spend.
  • If you want to establish a common identity across online and offline channels.

In order to participate for the Private Graph, the customer must be on Analytics Ultimate or have Audience Manager or have Target Premium and also be an Adobe Experience Platform customer. If you qualify for this as a client, subscribing to this should be a no-brainer for you given that no privacy policy updates need to take place. However, Device Co-op does give you access to a much larger amount of users which you wouldn't get with a Private Graph so you'll have to weigh your options on which one to choose.


Device Graph and Adobe Experience Cloud Solutions


In this section, I'll cover how to leverage some of the Adobe solutions I've used in conjunction with Device Co-op. Please note that I've only used Adobe Analytics and Audience Manager for customers participating in Co-op but there are use cases pertinent to Adobe Experience Platform, Adobe Target and Ad Cloud as well.

Adobe Analytics

Device Co-op is a natural fit for Adobe Analytics given the availability of the "People" metric which deduplicates the user count tied to multiple device and provides a true representation of a user as opposed to a device. This was introduced back in 2017 so it's been around for a while now and you can find more information about it here. Below is an example of what this looks like in a sample taken from the Adobe documentation.



The other service you can leverage (I've not used it yet) is Cross Device Analytics (CDA) which allows you to analyze cross device behavior in Analysis Workspace using Adobe Analytics data (needs a cross device report suite). This is not a default service and all the details and eligibility requirements are included in this Adobe Spark page.


Audience Manager

If you are a member of the Co-op and use Audience Manager, you get a lot of benefits primarily around prospecting and cross device frequency capping in offsite marketing. The first thing to do is to setup your Profile Merge Rule similar to how it's shown in the screenshot below taken from Adobe's documentation. 

The other thing which is very consistent with the screenshot is that your Co-op devices per Person count should be lower than the Person count of other data sources. Finally, you should expect to see a much larger count of users (potential reach) in the Co-op Person number compared to any other data source.


Adobe Experience Platform, Adobe Target,  Ad Cloud

Device Co-op also provides additional capabilities to Adobe Experience Platform for id stitching, Adobe Target for personalizing experiences across devices and to Ad Cloud for media uses cases around retargeting across devices. I haven't personally leveraged Co-op for AEPTarget and Ad Cloud so I'm unable to provide any additional context as I don't have much exposure to these three in the context of Device Co-op.

So, that was a high level overview of the Device Graph but I'll be writing more about it as I learn more about how it's used with other solutions of the Adobe Experience Cloud stack. Please feel free to share your feedback!

Saturday, February 8, 2020

Review Adobe Experience Cloud ID Cookies on Google Chrome 80

Web Browser Cookies have long been the lifeblood of the digital marketing ecosystem. A cookie is a tiny text file used by a web browser that typically captures non-intrusive information about a browser (indirectly a user) such as logged in state, user preferences and anonymized or encrypted ids etc. A cookie allows companies to measure browsing patterns (pages visited, products bought), remembering products added to a shopping cart or simply personalizing user experiences that match previous patterns and behaviors. 

As of December 2019, Google Chrome was the most popular web browser with a market share of 69%. Google recently released their newest Chrome browser version 80 which introduced clear guidelines around how cookies need to be set moving forward keeping in mind privacy regulations such as GDPR and CCPA. As an example, Safari now completely blocks 3rd party cookies from being set. Given that Google has advertising platforms which primarily leverage 3rd party cookies, there are still a few years left before Google completely phases out 3rd party cookies by 2022 which means we don't have a lot of time left. So, what does it mean for companies like Adobe which also rely on cookies for measuring user behavior using 1st party cookies and activating anonymous profiles on publishers using 3rd party cookies in the interim?

In this post, I will review changes made by the Adobe Identity Service team to address cookie setting requirements in Google Chrome 80.  I will refer to this article written by the Adobe Identity service team and validate changes made by Adobe primarily around the Experience Cloud ID Service cookies. So let's dive in!


What is all the Fuss About?


Here, I'll cover exactly was has been changed by Chrome 80 and I've made a simple decision tree to depict what the change is in regards to the new cookie guidelines. At a high level, 3rd party cookies need to be secure with a SameSite attribute equal to "None". On the other hand, cookies without a SameSite attribute will default to "lax". 

In the next two sections, I'll do a quick pre/post comparison between Chrome 79 and Chrome 80 and review the Experience Cloud demdex cookie set in both versions.


The World Before Google Chrome 80


I'm first looking at Chrome version 79 and I've visited an Adobe customer website using the Experience Cloud Visitor ID Service.

I'll primarily focus on the demdex which is our 3rd party cookie that is most susceptible to deletion after this change. I previously wrote about migrating to the Visitor ID service where I've covered some of the cookies set by the ID service in more detail. The first thing we see is an error in the developer console specifically calling out the demdex cookie and stating that it's set without the SameSite flag.

The World After Google Chrome 80


In this section, I will show how the Google Chrome error for demdex went away after I updated my browser version and look at the site customer website in incognito mode.

Looking at what the developer console shows for the same customer website, we can see that the change was made by the Adobe ID Service (demdex) server side and the error went away. Please note that it DID NOT require a Visitor ID service version upgrade as the change was made server side. Having said that, it's always advisable to upgrade your ID service library to the latest version. 

Now looking at how the demdex cookie is now set as shown by Chrome 80, we can see that the cookie has both the SameSite=None and Secure values set with a TTL expiration of 6 months.

What is the Recommendation for 1st Party Cookies on Safari?


In this section I want to talk about the importance of leveraging a CNAME tracking server to measure your website activity in Adobe Analytics which is more of an issue post ITP2.1 in Safari (slightly going off topic). This Adobe article covers how we can use a CNAME to set a new s_ecid cookie that extends the AMCV cookie expiration to 2 years instead of 7 days which Safari enforces today (see below).

Please note that this requires ID service version 4.3.0 + to take advantage of this change to extend your visitor expiration to 2 years instead of 7 days on Safari.

So, that's it! Hope found this helpful in understanding what changes were made by Chrome 80 and how Adobe is prepared to address any potential tracking issues as a result of this.

Monday, January 27, 2020

Adobe I/O Events: Webhooks and The Triggers API Call

Companies around the world are continuously evolving and finding creative ways to optimize and improve the user experience across numerous platforms. The latest buzzword in that shift is Serverless Computing which is slowly shifting the narrative by allowing you to deploy abstracted code to tackle specific use cases as opposed to setting up massive a Server infrastructure. Now, I'm not saying that the latter is going away but activating specific actions is what some companies are slowly moving towards without necessarily relying on 3rd party hosted solutions. One such use case where Serverless can help is Triggers which I wrote about last year.

I also gave an overview of Adobe I/O where I wrote about the four I/O components and I/O Runtime is at the helm of it all. In this article, I'm going to walk through the application of application of Adobe I/O Events in Triggers. I'm also going to cover how to setup webhooks on your machine (Mac in my case) and leverage a script the Adobe development team wrote to use your machine to "listen" for incoming Triggers. So let's get started.


Triggers Architecture with Adobe I/O


Below is a high level architecture diagram I created to show how the Triggers API would interact with the various Adobe I/O components primarily Runtime and Events. Please note that this is a subset of what all is possible with I/O and an overall Adobe I/O architecture diagram is still in the works which will show the complete picture. 

The steps till #7 in this diagram are almost the same as any regular Triggers lifecycle which I covered in my previous post so here, I'll cover everything from step 8 onwards. If some steps are not clear, don't worry as I will cover these in more detail later in the post. So, let's walk through these in more detail:

8a - A user subscribes to the Triggers API using Runtime by creating a webhook. A prerequisite is to subscribe to the Triggers API which is done in the Admin console (covered later).
8b - The webhook is registered in the Adobe Admin console which is validated by the Event Gateway.
9a - The Analytics Trigger is sent to I/O Events as soon as the trigger is executed (E.g. cart abandonment, some page is viewed etc.).
9b - I/O Events sends the Trigger JSON payload to the client which is collected at some endpoint leveraging Runtime via a POST request.
9c - A Runtime action is invoked  as a custom event to send the JSON payload to a 3rd party Email Service Provider (ESP). Please note Adobe Campaign will automatically receive the JSON payload (step 7).
10 - Runtime action is invoked to query data and receive the output.
11 - This output is sent to an SFTP location to Adobe Campaign or other ESPs via another action. 
12- ESP sends an email or SMS tied to the trigger payload and conditional logic.
13 - User clicks on the email and is routed back to the site.


Listen for I/O Events Via a Webhook


In this section, I'll cover some steps on how to setup a webhook using ngrok which is a tunneling software that allows you to convert your local site URL into a public facing URL. Please note that I'm not using the Runtime AIO CLI to do this but I have it installed on my machine and the instructions to install it can be found here

Setup ngrok


Download ngrok from https://ngrok.com/ and run the ngrok application from your Terminal or Command Prompt pointing to the folder where the application is located. In my case, it's located in the /openwhisk folder and I'm creating an ngrok URL tied to my localhost on port 3000 by executing the command ./ngrok http 3000. I'll cover why I chose 3000 later.

Once executed, you'll get the public facing URL for your localhost which is https://c926b37a.ngrok.io/ in my case. Please note that this URL is only public facing for 8 hours but I believe there's a paid version to make reserved URLs.

Create a Webhook "Listening" Server


In this section, I will execute some code written by the Adobe I/O Events development team (A big Thanks for them!) that I retrieved from the Adobe Github page which has additional instructions on how to set this up as well but the main thing is to install this package on your machine. The command to run is npm start (Node.js) once you've installed the package to run the app.js file.

Once it executes, it publishes an HTML page on port 3000 which allows us to create our webhook. Note that this is the same port on which I created my ngrok URL. On this page, I enter a path (trigger) which creates a webhook as https://c926b37a.ngrok.io/webhook/trigger that will "listen" for any incoming triggers.

Register Webhook in Admin Console


The first step is to create an integration with Analytics Triggers in the Adobe I/O admin interface. Follow these steps to create it.


Once the integration is complete, the next step is to click on the "Events" tab of your integration and register & test your newly created webhook. Note that I'm registering the same webhook created in the previous step which is https://c926b37a.ngrok.io/webhook/trigger.

Once you save it, a challenge will be sent as a GET request to this webhook to check if it's active.

If it's live, then you'll see the "Active" status in the admin console page which means your webhook is now ready to listen for Triggers.


Create and Test Your Trigger


For the purpose of this post, I've created a very simple Trigger Action to fire when a user visits my homepage called "sandbox-home" and visited my page a few times.


Aside from the Triggers UI, I can also see five Trigger events posted on the app.js page actively listening for the events. Based on my test, it took the between 20-30 seconds for me to receive the webhook (see below) and I got 5 triggers (POST requests).


Parse the Trigger POST Response


Finally, you can parse the trigger response by expanding the POST request. In the JSON response, we can see that there are two main components of the call where the first one has information about the timestamp, IMS org among others and the second one is the actual analyticsHitSummary payload that contains all the "enriched" data from Adobe Analytics. Please note that a valid use case for this would be to send this to a 3rd party Email Service Provider given that Adobe Campaign receives it automatically. Please note that the customer ID (used to link ACS with a customer ID) captured on your site needs to be sent in an eVar as part of the Payload.


BONUS: Send a Slack Message When a Trigger Occurs


Now, this section is just to show how to send a message to a Slack channel when a Trigger fires. Please note that this is probably not a good use case for Slack given the frequency of Triggers so some better use cases will be server maintenance notifications, code publishing notifications, new members added or removed, fraud detection among many others. I'll just use the Triggers API to show how it works:

Enable Incoming Webhook for Slack


In this step, we will integrate our webhook with Slack to send a Trigger response to a Slack channel when an event occurs. You can do that by going here which is the regular process. In my case, I went here and added this app to my Slack workspace and channel called #trigger-channel.

Once you complete the integration, you'll receive your Slack Webhook URL which looks like this: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

Once the integration is complete, you can send a POST request to the hook and post a custom message like I did via Postman. 

Once it's done, you'll see a bunch of messages stating the the integration is setup including a "Hello World" message which I sent. 

Next, I'll show you how to enable Slack to receive your Triggers webhook.

Modify Code In app.js


Modify the app.js file and update your webhook URL and channel name.

Next, we'll modify the Triggers payload to customize what we want to send to our Slack channel. In my case, I want to display my ECID and URL which I've slightly modified in the script written by the Adobe I/O team.

Monitor Triggers in Slack


Finally, we can see that we got 5 Trigger responses in our Slack channel as well like we saw in the Triggers UI and the ngrok "listening" page. Again, not sure if this a good use case for Slack but you can use the model for other use cases which are more pertinent. 

So, that was it! Hope you found this helpful for your own use case on how to use I/O Events. Are you using Adobe I/O in any form today?