# Working w/ Warehouse Data

### About this export

| Field | Value |
| --- | --- |
| **content_type** | lesson |
| **platform** | contentstack-academy |
| **source_url** | https://www.contentstack.com/academy/courses/data-insights-data-ingestion-profile-construction/data-insights-course-3--working-with-warehouse-data |
| **course_slug** | data-insights-data-ingestion-profile-construction |
| **lesson_slug** | data-insights-course-3--working-with-warehouse-data |
| **markdown_file_url** | /academy/md/courses/data-insights-data-ingestion-profile-construction/data-insights-course-3--working-with-warehouse-data.md |
| **generated_at** | 2026-11-09T09:53:49.935Z |

> Part of **[Data Ingestion & Profile Construction](https://www.contentstack.com/academy/courses/data-insights-data-ingestion-profile-construction)** on Contentstack Academy. **Academy MD v3** — structured for retrieval; no quiz or assessment keys.

<!-- ai_metadata: {"lesson_id":"10","type":"video","duration_seconds":579,"video_url":"https://cdn.jwplayer.com/previews/u1mD3rGg","thumbnail_url":"https://cdn.jwplayer.com/v2/media/u1mD3rGg/poster.jpg?width=720","topics":["Working","Warehouse","Data"]} -->

#### Video details

#### At a glance

- **Title:** 18-data-insights-warehouses
- **Duration:** 9m 39s
- **Media link:** https://cdn.jwplayer.com/previews/u1mD3rGg
- **Publish date (unix):** 1752878424

#### Streaming renditions

- application/vnd.apple.mpegurl
- audio/mp4 · AAC Audio · 113588 kbps
- video/mp4 · 180p · 180p · 135498 kbps
- video/mp4 · 270p · 270p · 148923 kbps
- video/mp4 · 360p · 360p · 162108 kbps
- video/mp4 · 406p · 406p · 170215 kbps
- video/mp4 · 540p · 540p · 200907 kbps
- video/mp4 · 720p · 720p · 245465 kbps
- video/mp4 · 1080p · 1080p · 382520 kbps

#### Timed text tracks (delivery)

- **thumbnails:** `https://cdn.jwplayer.com/strips/u1mD3rGg-120.vtt`

#### Video transcript

So, we talked about integrations that pull from marketing tools. We talked about all their APIs, our JavaScript tag, which uses our APIs. The thing that we haven't touched on that I think is pretty easy to demo, it certainly can be a bigger conversation to go into the weeds. But the other source of data that is super common to pull in is from your warehouse. So we have integrations in the data pipeline, if you just want to stream the entire table in and not have any filtering from your warehouse, you can do that through the connection just like we did MailChimp. But we have a special tool called Cloud Connect that allows you to actually connect to your warehouse. We support all of the major warehouses, BigQuery, Snowflake, Redshift, et cetera. What it allows you to do is create a connection, which is very similar to what we did with MailChimp, so I won't walk through that. Like it's a BigQuery, we just use a JWT to get authorization into BigQuery. But what it allows you to then do is build data models, which is essentially a SQL query against that particular table or set of tables inside of your warehouse to pull that data in uniquely to the profile. It goes through, and this is why we'll have a bigger conversation on what it actually means to the profile and how it works, but it maps things in a pretty unique way in that it doesn't go through a data stream. It doesn't necessarily have to adhere to the mapping and the sort of those rules. It creates its new own set of fields that also go away if you lose access to the data. So this comes up a lot when customers want to add scores or information to a profile, but they don't want to create a duplicate and copy and stream it in and go through all of the inherent kind of risks there. They just want to plop something on a profile and kind of like override some of those settings. It allows you to go in, in this sample BigQuery instance, it'll actually pull up a SQL editor. I have a very simple query that I wrote. You can just paste it and say, I want to select everybody from the sample data set with email first name, last name, and an average annual revenue from sample customers. You can test the query. It'll actually query that, in this case, BigQuery in real time. And then as you connect it, you can actually then describe how you want to map that to a profile. So we'll just say BQ test. The only thing really you have to choose is the primary key. So from my data set, I want to map the email that I just selected. Again, that's the only kind of context. You still have to tell it how you're going to map this Cloud Connect data, this warehouse data to a profile. So I want to merge it based on the email address. I want to merge that with the email field. And then you can choose optionally if you want to pull in additional information. So I want to add first name, last name, and average annual revenue. With Cloud Connect, because it's less of a real-time thing, more of a query-based thing, you can then choose the cadence of how often you want that to run. For tests and demo, I always do an hour. In reality, you probably don't want to just spam your warehouse instance with these big expensive queries every single hour. So most customers are going to do 24 hours or 48 hours or whatever it may be. But you have that. You can flag if you want to create net new profiles from that data. If you don't check this box, it's only going to map to the profiles that exist and never create net new ones. Because our sample database isn't built of Game of Thrones characters, I'll create new so that it creates those profiles. Then ultimately, you create this data model, and it's going to go through, query that BigQuery database on that cadence that we described. And then ultimately, those profiles will come in to the UI. I don't know how long that will take. So I think in our next session, I'll be sure to show you what that data looks like on a profile because it looks a little bit different. All of the segmentation and activation capabilities are exactly the same. But I just wanted to touch quickly just to introduce the idea that's the final piece of where data can come from. The Cloud Connect product represents a little bit of a different method for getting data into Linux. All of the other methods that we talked about, the JavaScript tag collection, the APIs, all of our background jobs, all of that kind of stuff uses our streaming pipeline. So it goes into a stream, a stream maps to a field, fields ultimately show up on the profile. Cloud Connect, just to kind of re-cover this part, is quite a bit different, actually, in that it doesn't use our streaming pipeline to actually get data onto the profile. It has a whole different mechanism that we can talk at length at. At some point, Eric can go into details there, but it essentially bypasses that streaming pipeline and injects the results of that query directly onto the profile. It's really useful for a few different reasons, but the one context that it comes up often is around sort of security and control. So think about like a situation where customer A wants to share a subset of data with one of their partners, with an agency, with another customer, whatever it may be, but they don't want to just give access to the raw data, something that they can like copy and own forever. So all of the warehouses have a different kind of methodology for how you can share and unlock that capability. In BigQuery, you essentially can give access to a specific dataset, and then you don't have to necessarily expose all of the raw data. With how Cloud Connect works, where it doesn't stream that data in, it's not creating a sort of like hard copy that gets written to our system, that gets backed up in our files, and it's creating kind of a less persistent temporary store for that data. If customer A wants to unshare essentially that data from their partner, from the other customer, whatever it may be, and they then lift that access, so they prevent somebody from actually being able to query that database, the next time that query runs, it'll actually clean up that system all the way through, so you don't have that kind of like legacy data that's in that stream system and all that kind of stuff existing. So it comes up often when security or data control, data access, is a key part of that conversation. And it's because, like I said, it doesn't actually stream the data to Lytx in the exact same way. So to just quickly kind of recap on Cloud Connect, it's under data pipeline, the same place that all of our jobs and all the other sort of collection profile sort of building is. Within Cloud Connect, you have the idea of connections, which is just that connection to the database. We won't rehash there. You have the data model, which is essentially the query that's going to run. In our case, in the last conversation, we built this sample query. It just pulls in a set of sample users, first name, last name, email, customer type, and then just an example of a score, for instance, that would be maybe in your warehouse. The thing that we didn't totally cover was how you then get the Cloud Connect data, this warehouse query into Lytx to store it on a profile so that it functions essentially fundamentally like one of our normal attributes. There's a publishing process in this. So when I hit next, but I think we might have briefly touched on this, if I recall, but we didn't actually complete it. And then we definitely didn't show it on the profile. So with Cloud Connect, you don't have to have everything mapped. You don't have to have all of the attributes configured in the same way that the streaming pipeline is. The only question that you have to answer is essentially how to map this data to a single profile. So you have to essentially pick the key from your data that you're querying from, say BigQuery, and what key you want to be able to write it to, so which identifier inside of Lytx you want to associate that data with. So in this case, in this one that I've already pre-configured, we're basically just saying in this query, there's a bunch of stuff, but all we want to do is we want to find anybody that matches on email. And if they match on email, we're going to append this information to that profile. The thing that is unique about Cloud Connect, if I go to a profile, for instance, that I think I had pulled up, yeah. So this one is one of the records that's in that sample data set. They have a profile, just like any other user, regardless of where that was generated from. But if you scroll down and see the data that came in for that particular data model on their profile, you're going to see a few different things. One, you'll see the raw attribute that we pulled in, first name, last name, but it's going to be independent of the other first name, last name fields that are already in the schema. And then you see this unique, and this is actually the more useful part of this particular thing, is there's this unique membership attribute that now gets added. So back to that example of like, I'm Nike.com and I'm sharing data with a partner and I want to give them access to everybody that has a high propensity to buy women's running shoes or whatever it may be, but I don't want to give them all of the data that I needed to use to pull this score to build that list. I just want to give them sort of that Boolean yes, no. That's where this membership flag can come in and that you don't have to have access to all of the information in order to make the calculation. You're just essentially pulling this information in temporarily as long as you have access. And then when you go to build a segment, it functions full scale, just like all of the other attributes in the system. So you can mix and match. There's no sort of limitations there, but like the one thing to just kind of be aware of and know is the use case of where this particular method for pulling data in from a warehouse is super useful, is around that sort of access control. I want to be able to pull things away. I don't want that data to persist. I don't want to store it somewhere. Whereas the streaming method, which also has a warehouse connection, or if I just want to pull everything from a particular table or whatever it may be, you can use our kind of back end integration.

#### Key takeaways

- Connect **Working w/ Warehouse Data** back to your stack configuration before moving to the next module.
- Capture one concrete artifact (screenshot, Postman call, or code snippet) that proves the step works in your environment.
- Re-read the delivery versus management boundary for anything you changed in the entry model.

## Supplement for indexing

### Content summary

Working w/ Warehouse Data. Working w/ Warehouse Data in Data Ingestion & Profile Construction (data-insights-data-ingestion-profile-construction).

### Retrieval tags

- Working
- Warehouse
- Data
- data-insights-data-ingestion-profile-construction
- lesson 10
- Working w/ Warehouse Data
- data-insights-data-ingestion-profile-construction lesson

### Indexing notes

Index this lesson as a primary chunk tagged with lesson_id "10" and topics: [Working, Warehouse, Data].
Parent course slug: data-insights-data-ingestion-profile-construction. Use asset_references URLs as thumbnail hints in search results when present.
Never surface LMS quiz content or assessment answers from this file.

### Asset references

| Label | URL |
| --- | --- |
| Video thumbnail: Working w/ Warehouse Data | `https://cdn.jwplayer.com/v2/media/u1mD3rGg/poster.jpg?width=720` |

### External links

| Label | URL |
| --- | --- |
| Contentstack Academy home | `https://www.contentstack.com/academy/` |
| Training instance setup | `https://www.contentstack.com/academy/training-instance` |
| Academy playground (GitHub) | `https://github.com/contentstack/contentstack-academy-playground` |
| Contentstack documentation | `https://www.contentstack.com/docs/` |
