MinusOneDB Docs

Appearance

Event capture ​

Our event capture solution should have everything you need to understand user behavior in any context. The samples below illustrate common patterns for identifying, storing, and querying events. You can create events when a customer adds an item to a shopping cart, swipes right on a mobile app, leaves your site or anything else of interest to you. Or you can apply the same patterns to capture data from devices for monitoring or telemetry purposes.

To get started by instrumenting a website or other JavaScript app continue to the next section. If you would prefer to explore using simulated traffic jump to the Event Simulator setup instructions. This can be useful for testing before your website goes live or for testing large volumes of events.

Instrument Your Website ​

See how users interact with your web application.

Setup Time: 20 minutes

Run Time: Immediate

This tutorial shows how to instrument your web application using MinusOneDB. This will allow you to understand how users are interacting with your website in great detail. Once you have completed the steps below, user interaction events will be sent from your website to your minuseondb environment. These events can be queried in near real-time to understand how users are interacting with your site. This tutorial will also guide you through some common customization paths to further enrich the data you can capture.

Prerequisites ​

Step 1: Download and unzip the instrumentation sample.

Step 2: Set up the m1 client if you have not already done so.

Step 3: You will need a MinusOneDB environment:

  • If you have never used MinusOneDB before, you can set up a free trial environment and then create an environment using the events.json environment template file available in your download.

  • If you have a MinusOneDB account but have not set up an environment yet, create an environment using the events.json environment template file available in your download.

  • If you already have your target environment follow the instructions below to instrument your website and begin capturing data.

Instructions ​

Step 1: Copy the instrumentation.js, M1Events.js, and lib/jquery-3.6.0.min.js files from the instrument/src sample directory into the same folder as your web application's other JavaScript files. There is no need to copy the jQuery file if your site already uses it.

To make a new user with appropriately restrictive rights:

m1 envName-accountName user/add -username instrument -password ChangeMeNow! -rights '["publish"]'
curl https://envName-accountName.minusonedb.com/user/add \
-d 'username=instrument&password=ChangeMeNow!&rights=["publish"]' \
-H "m1-auth-token: $adminToken"

To restrict the rights of an existing user to only publish:

m1 envName-accountName user/update -username instrument -rights '["publish"]'
curl https://envName-accountName.minusonedb.com/user/update \
-d 'username=instrument&rights=["publish"]' \
-H "m1-auth-token: $adminToken"

Step 2: Set the environment name and user account in the "REQUIRED PARAMETERS" section of the copied instrumentation.js file. Typically this account should be a system account not used for any other purpose. This account should only have the publish right.

Step 3: Source the JavaScript files copied above by adding the following lines to the head of your html file (or other location where you import JavaScript files); this is often in an index.html file.

<script src="js/jquery-3.6.0.min.js" type="text/javascript" charset="utf-8"></script>

<script src="js/instrumentation.js" type="text/javascript" charset="utf-8"></script>

<script src="js/M1Events.js" type="text/javascript" charset="utf-8l"></script>

Note: You may need replace the js/ directory in these examples with the directory from which your application sources JavaScript files. It can be omitted if the files are added to the top level directory.

Add localhost.somethingunique to the cors domain whitelist:

m1 envName-accountName system/cors/add -domains localhost.somethingunique
curl https://envName-accountName.minusonedb.com/system/cors/add \
-d "domains=localhost.somethingunique" \
-H "m1-auth-token: $myToken"

Step 4: The sample instrumentation.js file contains reasonable defaults for instrumenting a single page web app; sessionStart events are recorded when a user first comes to a site and view events are created as users navigate via hyperlinks. instrumentation.js contains some other sample event capture patterns that you may want to adopt or customize for your own needs.

Step 5: You will need to add your website's domain to your environment's CORS domain whitelist. You can add as many domains as needed. If you are testing on localhost, add a unique entry to your hosts file that resolves to localhost, i.e., 127.0.0.1 localhost.somethingunique and then add that as whitelist entry using one of the commands below. We encourage you to add the entry to hosts rather than directly adding localhost as a whitelisted domain to prevent any chance of running into a CORS related vulnerability.

See the 5 most recent events:

m1 envName-accountName query -store index -q '*' -sort "_m1.receivedUTC desc" -rows 5
curl https://envName-accountName.minusonedb.com/query \
-d "store=index&q=*&sort=_m1.receivedUTC desc&rows=5" \
-H "m1-auth-token: $myToken"

See events by state:

m1 envName-accountName query -store index -q '*' -json '{
  "facet": {
    "by_state": {
      "field": "_m1.geo.subdivision_1_iso_code",
      "type": "terms"
    }
  }
}'
curl https://envName-accountName.minusonedb.com/query -d 'store=index&q=*&json={
  "facet": {
    "by_state": {
      "field": "_m1.geo.subdivision_1_iso_code",
      "type": "terms"
    }
  }
}' -H "m1-auth-token: $myToken"

Step 6: Open up your site in a browser; if your site was hosted at localhost, be sure to use the hosts entry you created above in place of localhost (i.e. http://localhost.somethingunique:8080/index.html rather than http://localhost:8080/index.html). As you navigate around your site, events will flow into your MinusOneDB environment; you can query your events using our reporting application or by using the sample queries below.

Troubleshooting Tip: If the queries above are not returning results, open your browser's console and check if there are any errors. Doublecheck each of the steps above, making sure your site has loaded the custom JavaScript, your user credentials are correct and there are no other obvious errors. If you get stuck, reach out to us.

Congratulations! You have now successfully instrumented your website. You are capturing web events as they occur and you can query this data in near real-time. Continue reading for some guidance on taking this to production.

Next Steps ​

Before deploying these changes to your production web site:

Add minusonedb.com to the cors domain whitelist:

m1 envName-accountName system/cors/add -domains minusonedb.com
curl https://envName-accountName.minusonedb.com/system/cors/add \
-d "domains=minusonedb.com" \
-H "m1-auth-token: $myToken"

Step 1: In instrumentation.js, set the async parameter to true. This allows the MinusOneDB servers to optimize the storage of events in your production environment.

Step 2: Add your production website domain to your MinusOneDB environment's cors whitelist. Replace minusonedb.com with your domain as in the examples below.

Step 3: There are a number of sections in instrumentation.js that may be of further interest. In particular, consider adding heartbeat events to gain a much richer time series based perspective of your users' behavior.

Heartbeats ​

See how users interact with your web application over time.

Setup Time: 5 or 25 minutes (depends on whether you have worked through the instrumentation sample).

Run Time: Immediate

Heartbeats are periodic events that record that a user is still present. They allow you to understand how long a user stays on your site and when they leave it. Heartbeats directly allow you to calculate the number of concurrent users on your site that can then be broken down by geography or individual page. In addition to the notion of presence, any attributes available on heartbeat events allow for a time series perspective that is not possible otherwise. What pages on your site do users spend the longest time on? What is the most frequently viewed viewed page right before a user leaves your site? Heartbeat events can help answer these and many other questions.

The volume of heartbeats events will typically dwarf that of user interaction events. While we have found a great deal of benefit in heartbeat events at a 5 second frequency, some customers will want to optimize for cost. This can done by lowering the heartbeat frequency or by not persisting heartbeats at all and instead using them only to calculate user departure. See configuring heartbeat events with a session store or continue reading to persist all heartbeat events. We strongly encourage persisting heartbeats to start. The persisted heartbeats will be more cost-effective until you have collected 1-2 million hours of recorded user interactions. By then you will have a better understanding of the value of these and other events to your business.

Prerequisites ​

Step 1: Work through the steps of the Instrument Your Website sample above.

Instructions ​

See the 3 most recent heartbeat events:

m1 envName-accountName query -store index -q "event.type:heartbeat" -sort "_m1.receivedUTC desc" -rows 3
curl https://envName-accountName.minusonedb.com/query \
-d "store=index&q=event.type:heartbeat&sort=_m1.receivedUTC desc&rows=3" \
-H "m1-auth-token: $myToken"

Step 1: In the instrumentation.js file you edited above, configure an appropriate HEARTBEAT_INTERVAL_MS value and uncomment the line that saves all heartbeats.

Step 2: Open up your site in a web browser. Heartbeat events will periodically be sent to your minusonedb environment. You can query your events using our reporting application or by using the sample queries below.

Heartbeats on a Budget ​

Capture heartbeats in ephemeral storage to understand concurrent user metrics while reducing costs for large sites.

Setup Time: 10 or 30 minutes (depends on whether you have worked through the instrumentation sample).

Run Time: Immediate

While heartbeat events offer many benefits, their number will typically exceed the number of non-heartbeat user interaction events. You may not wish to persist a high volume of potentially similar heartbeat events, while still desiring analytics around user presence. Rather than storing heartbeats in a datalake and indexed store, this sample shows how a client can send heartbeats to a session store and have a separate session flusher process determine when sessions have timed out. In other words, this sample allows you to record only the last (and most analytically interesting) heartbeat for a given session and foregoes persisting the intervening ones. Note that the added cost of the session store is roughly the equivalent in price of storing an additional 1-2 million hours of user interactions; at smaller volumes persisting heartbeats will be more cost-effective.

Prerequisites ​

Step 1: Work through the steps of the Instrument Your Website sample. You can save a step if you use the eventsSessionStore.json environment template file rather than the events.json described in that sample.

*Step 2: Download and unzip the session flusher sample.

An example addSessionStore.json to add a new session store with name stash:

{
    "environment": {
        "stores": [\
        ],
        "sessionStores": [{\
            "store": "stash",\
            "shards": 1,\
            "replicas": 1\
        }]
    }
}

If you have already created an environment from events.json you can modify it to look like the eventsSessionStore environment by performing the following operations. These commands will add a session store, create a flush user with appropriate permissions, and allow the instrument user to write to the session store.

m1 orchestrate addStores envName-accountName envKey addSessionStore.json

m1 envName-accountName user/add -username flush -password "change this password" -rights '["publish", "sessionUpdate", "sessionQuery"]'

m1 envName-accountName user/update -username instrument -rights '["publish", "sessionUpdate"]'

Step 3: Install and configure python if you have not already done so.

Instructions ​

Configure the Session Flusher

Sample config.json.sample for reference:

{
    "server"   : "server name (ex: https://live-m1.minusonedb.com)",
    "session"  : "session name (ex: stash)",
    "username" : "username",
    "password" : "password",

    "timeoutSeconds" : "time (in seconds) before the flusher considers a session to have timed out;
                      you'll generally want this to be 3-5x your heartbeat period."
}

Step 1: In the flusher/src directory, copy config.json.sample to config.json. Edit your environment, environment credentials, session store, and data store in config.json

Step 2: Run python3 -u flusher.py config.json

Step 3: You should see some output saying that there is "nothing to flush". flusher.py checks every few seconds to see if there are timed out sessions.

Step 4: Leave flusher.py running; you will come back to it after configuring instrumentation.js

Configure instrumentation.js

Step 1: Open the instrumentation.js file you have already edited and find the "Using a session store with heartbeats" section.

See heartbeats in the session store:

m1 envName-accountName session/query -store stash -q "*"
curl https://envName-accountName.minusonedb.com/session/query \
-d "store=stash&q=*" \
-H "m1-auth-token: $myToken"

Step 2: Configure the session store name, configure the heartbeat frequency, and uncomment blocks of code as directed in instrumentation.js

Step 3: Open up your site in a web browser. Heartbeat events will be periodically sent to your configured session store.

Step 4: Query for heartbeats in the session store. You should see a document representing your active session.

  • Troubleshooting Tip: If your session store query returns no results, make sure there are no JavaScript errors and doublecheck the configuration steps above.

See persisted sessionTimeout events:

m1 envName-accountName query -store index -q "type:sessionTimeout" -rows 5
curl https://envName-accountName.minusonedb.com/query \
-d "store=index&q=type:sessionTimeout" \
-H "m1-auth-token: $myToken"

Step 5: Now close the browser that was visiting your site -- heartbeats will no longer be sent to the session store. Navigate back to where you ran flusher.py. After the appropriate amount of time has elapsed (configurable via the timeoutSeconds parameter in the flusher.py config file), you will see a "Preparing to flush" message in the flusher.py output signifying that flusher.py found your session that has timed out.

Step 6: Additionally, you will see a sessionTimeout persisted event recording the time your session timing out. You can use the sample query below to retrieve these events.

Congratulations! You now have system that allows you to determine the number of users that are actively using your site at any point in time at a minimal cost.

Event Simulator ​

Generate synthetic web events to familiarize yourself with the MinusOneDB event capture solution rather than instrumenting your own website.

Setup Time: 20 minutes

Run Time: Immediate

This python sample simulates user interactions on a web page. Use it to quickly create some test data to gain familiarity with MinusOneDB concepts. Run a number of simulator processes concurrently to simulate high traffic. Use in concert with the session flusher to test your own session flushing criteria.

Prerequisites ​

Step 1: Download and unzip the event simulator sample.

Step 2: Set up the m1 client if you have not already done so.

Step 3: You will need a MinusOneDB environment:

  • If you have never used MinusOneDB before, you can set up a free trial environment and then create an environment using the events.json environment template file available in your download.

  • If you have a MinusOneDB account but have not set up an environment yet, create one using the eventsSessionStores.json environment template file available in your download.

  • If you already have your target environment follow the instructions below to run the event simulator.

Step 4: Install and configure python if you have not already done so.

Instructions ​

config.json.sample for reference:

{
    "server"   : "server name (ex: https://live-m1.minusonedb.com)",
    "session"  : "session name (ex: stash).  Set this param to null to simulate persisting heartbeats",
    "username" : "username",
    "password" : "password"
}

In the simulator samples directory, copy config.json.sample to config.json. Edit your environment, environment credentials, session store, and data store in config.json.

Simulate Web Events ​

Run this command to execute the python simulator:

python3 simulateClient.py config.json

The script simulates actions users perform on a webpage: arriving at your site, navigating to pages, and leaving the site. Heartbeat events are also simulated; they will be sent to a session store if you have configured one. Otherwise heartbeats will be archived in your environment like all other events.

Simulate Many Web Events ​

See heartbeats in the session store:

m1 envName-accountName session/query -store stash -q "*"
curl https://envName-accountName.minusonedb.com/session/query \
-d "store=stash&q=*" \
-H "m1-auth-token: $myToken"

See the 5 most recent events:

m1 envName-accountName query -store index -q '*' -sort "_m1.receivedUTC desc" -rows 5
curl https://envName-accountName.minusonedb.com/query \
-d "store=index&q=*&sort=_m1.receivedUTC desc&rows=5" \
-H "m1-auth-token: $myToken"

See events by state:

m1 envName-accountName query -store index -q '*' -json '{
  "facet": {
    "by_state": {
      "field": "_m1.geo.subdivision_1_iso_code",
      "type": "terms"
    }
  }
}'
curl https://envName-accountName.minusonedb.com/query -d 'store=index&q=*&json={
  "facet": {
    "by_state": {
      "field": "_m1.geo.subdivision_1_iso_code",
      "type": "terms"
    }
  }
}' -H "m1-auth-token: $myToken"

To simulate N concurrent users interacting with a webste, run the following:

Step 1:./runSim config.json N

Step 2: Running tail -f *.log will give you a view into what is happening with your simulated clients.

Step 3: When you are done simulating, you can stop the running simulateClient processes by running ./stopSim

Step 4: (If using a Session Store) Query for heartbeats in the session store. You should see a document representing your active session.

Step 5: You can query persisted events using our reporting application or by using the sample queries below.

Was this page helpful?
Suggest an edit

© 2021-2026 MinusOne, Inc.