Product Management Insight

As Analysts working in Adobe Analytics or any other system, we want to analyze data without any restrictions or caveats around volume. Analysts who work with Adobe Analytics must've heard about the dreaded "Low Traffic" bucket which shows up when a report in the UI has more than 500,000 unique values per month (eVar/prop etc.). Here's how it shows up in the report.



A customer recently asked whether they could upload over a Million rows of classification data in Adobe Analytics tied to a primary user ID stored in an eVar and leverage the UI for analysis and segmentation. We discussed the implications of uploading this amount of classification data into Adobe Analytics and even considered Audience Manager. I recently wrote about the impact of classifications on segments for a different use case. In this post, I will cover some of the pros and cons of uploading over a million rows of classification data into Adobe Analytics and onboarding records into AAM for deeper analysis.

Pros and Cons of Classification Data (Over 5ooK Rows)

Adobe Analytics UI

• Uploaded classification data is retroactive
• There's no extra cost in uploading classification data
• Recommended tool for deeper data analysis and segmentation of classified dimensions (below 500K rows)

Cons

• Classification data over 500K rows might take days or weeks to catch up and may delay analysis
• Classification data in the UI AND segments tied to it will be subjected to the 500K unique limit
• Classification data is also subjected to hash collision where some IDs may show 2 or more Unique visitors in the UI as explained here. 
• Values over the “Low Traffic” bucket do not get classified. Data Warehouse does not have this limitation so if data can be exported out, we won’t run into the limit.

Audience Manager UI

• Onboarded data is retroactive IF data is segmented in AAM and an ID sync is in place
• Large amount of data can be handled without any volume constraints
• Data is uploaded to AAM quicker (24-48 hours) than how long Analytics classification  over 500K rows will take (assuming there are no errors encountered during onboarding)

Cons

• Limited ability for reporting and segmentation in the AAM UI compared to Analytics. The UI will show overall uniques based on segmentation but will not allow you to slice and dice the data like Analytics can. An example is lack of sequential segmentation which you get in Adobe Analyics
• Onboarded data/segments will only apply to users who visit the site AFTER the data has been onboarded when AAM segments are shared with Analytics and analysis is done there. I cover this in a post I wrote about last year.
• Each row of data will be charged as a separate server call
• Data is mostly available in the form of Unique Visitors and not as Page Views or Visits

As you can see, there's no clear winner when it comes to classifying large amount of data for analysis in either UI. So to summarize, the Adobe Analytics UI is better suited for deeper analysis and segmentation (data under 500K rows per report) whereas Audience Manager is better suited to handle large amount of data with segmentation capabilities suited more for activation of audiences (Cookies and Mobile device IDs).

Potential Options and Alternatives

• Leverage Adobe Data Warehouse: It's not perfect but you can leverage Adobe Data Warehouse to export data and run segmentation as you won’t run into the same limits as the UI.
• Reduce the Cardinality: You can split dimensions with high cardinality into separate variables.
• Increase the uniques exceeded limit "within reason": You can contact Customer care to increase the uniques limit from 500K to up to a Million. This will work for anything between 500K-1 Million but not for anything over that.
• New- Customer Journey Analytics: CJA is a newly added offering which leverages AI/ML models and much larger datasets from Adobe Experience Platform. This offering will remove all limitations around uniques exceeded and will be added to Workspace to allow for easy analysis of big datasets.

Again, there's no perfect solution to tackle this issue currently but the new feature of Customer Journey Analysis aims to fix this issue which is exciting. I hope this post was helpful to give you an understanding on some of the implications of uploading large amount of data into Analytics but all is not lost and there are some potential ways to avoid this.

At the start of this year, I wrote an article about Audience Manager on which I'm going to structure this article. The only difference is that this article will cover some random facts about Adobe Analytics which are not commonly known or are rarely discussed. One of my co-workers Adam Waddington has also shared his experience on an issue he faced with a client which is covered in the first bullet point followed up by some other facts which I've shared.

1. Segments Are Tied To SAINT Classified Dimensions

This is based on a recent issue which Adam faced while deleting a classified column as part of a new SAINT upload where segments mapped to one of the deleted columns stopped working. Below is a verbatim writeup of what Adam sent me and I hope you will find what he shared useful. Thanks Adam!

• DELETING a classification in AA removes that variable and its associated Div data from the backend. Any segment which leveraged the classified variable linked to that Div will then become ‘Undefined’ and segments will not function as intended.

• EDITING or renaming a classification in AA has no effect on the Div itself. However, any segment which uses that modified classification will now show the updated name provided.

• ADDING a classification will always default to incrementing by +1 on the highest Div# in the system, even if a lower Div# was deleted. For example, deleting Classification 2 from the above table and subsequently adding a new Classification will create Div4; Div2 (Classification 2) will still not exist. In order to add Div2 (in this example), the user must delete all Classifications below 1, add Classification 2 again, and re-upload the desired SAINT tables.

2.  s.transactionID Can Be Used To Reconcile Missing Orders

Late last year I wrote an article about data variance between Web Analytics tools and mentioned that there is a way to reconcile missing eCommerce orders on the purchase page with the help of Adobe engineering.  This can be accomplished by passing s.transactionID on a page prior to a confirmation page and data can be reconciled by engineering. The trick is to pass the s.transactionID right before the purchase page on the page (client side) as well as in the backend (server side). 

This is not straightforward as it will require additional support from your developers but our engineering team can take the two transactionIDs, reconcile them in the backend then backfill any missing orders. Please note that there will be an additional cost to work with engineering as well so ONLY do this if it's absolutely necessary.

3. Sending Historical Data To Adobe Analytics Is Possible BUT..

This is accomplished by passing over historical attributes (eVars, props, events etc.) along with the timestamp variable (s.timestamp) but the main issue is out of order nature of events being sent in as it can affect the sequence of events and attribution of eVars. The only use case I've seen for this is backfilling historical or missing data and should ONLY be done when you have no other option. I've included a sample below using a GET request via the Data Insertion API method. 

Notice that the timestamp is for a previous time frame. I'd also suggest changing the timestamp setting in your report suite to optional for best results especially if you're using a global setup capturing both mobile app and desktop data.

http://(tracking server)/b/ss/(rsid)/0?AQB=1&pccr=true&g=none&ts=1559335477&products=;testvarn;1;5.95&events=event1&pageName=testPN&mid=5670954657508492467610283427418697012&AQE=1

4. Possible To Send Negative Revenue In The Products String

One other rare scenario is where your client side code accidentally captures a really large order value which messes up your overall Revenue metric. This happened recently for a client where they accidentally sent over a Million dollars for a single SKU. There is a way to fix that by sending an equivalent negative revenue amount for the same SKU for which you accidentally sent the wrong amount. I've demonstrated it using a small amount of $400 but the concept is the same.


The first screenshot shows that we "accidentally" sent a revenue of $400.
5. tntA Variable Can Be Sent To Tie Analytics With Target (A4T)

This is from a recent test I did with a client while testing A4T where Analytics is deployed server side but Target is deployed client side. The data is sent to Adobe Analytics via Data Insertion API and the challenge was that there was no concept of a Supplemental Data ID matching Analytics and Target which is used to tie Analytics with Target client side. 

In this case, the fix is to send the tntA variable (captures A/B test variant information) which can be passed as a query string parameter both on page view and click events. This ensures that client side Target matches up with Adobe Analytics deployed server side and populates the A4T reports. Below is a sample (event) snippet you can send to test this which will populate the Analytics for Target reports in Adobe Analytics. Note that we've added pe=tnt to denote that it's a click event other for page views, you'll simple pass the tnta param without pe=.

http://(tracking server)/b/ss/(rsid)/0?ts=1559335477&products=;testpev;1;5.95&events=event1&pageName=testpev&mid=5670954657508492467610283427418697012&pe=tnt&tnta=291105:0:0|1

6. Submit Expected Traffic Spikes Well In Advance

A lot of businesses have expected traffic spikes each year during seasonal events such as Thanksgiving and it's quite common for them to schedule traffic spikes for their Adobe Analytics report suites. However, I've seen a lot of cases where these spikes are scheduled quite late which leads to delayed data processing of their data. 

To avoid that, it's better to schedule your expected traffic spikes well in advance to avoid any data processing delays or issues. As an example, expected spikes above 25,000,000 server calls should be submitted one or more months in advance which gives Adobe enough time to allocate any necessary hardware. Here's a summary of how much lead time is needed by traffic volume.

So that's it! Hopefully you found some of these uncommon scenarios pertaining to Adobe Analytics helpful.