Environment API

The services documented below let you interact with one of your MinusOneDB environments. All of the service endpoints described can be accessed via conventional HTTP clients (curl, postman) or programmatically in your desired language or framework. We offer the m1 client which simplifies credential management and environment interaction.

Services use typical HTTP response codes.

2xx: The request operation worked as as expected.

401: The token used for the request was invalid.

403: The user associated with the token used for the request does not have sufficient rights to perform the requested operation.

500: The request could not be completed

Authentication and Access

/auth

Log in and retrieve token for service access. By default, tokens must be used from the machine they were generated and will expire after a user defined time.

POST/auth

Base URL

username *

password *

ip

ttl

• rights: none, admin if ip or ttl is set
• verbs: POST

Parameter	Type	Required
username	String	Yes
password	String	Yes
ip	String	No. Used to log in on behalf of a user at specified ip address. Users with admin rights can use "..." here to generate a token that can be used from any ip address.
ttl	long	No. Will set a custom expiration time in ms. The default ttl is defined by the token-expire-ms system parameter.

curl

curl https://test-m1.minusonedb.com/auth \
-d "username=admin&password=passphrase"

m1 client

m1 auth test-m1

▶200 OKAuthentication token

"eFRxichv7cTeqks99XD+iNBqR3LlgILIZ3nhKieS8c8qajV9+JcNqSqZfngvS+gJM3IBKnvI2Crd66WOIswtlr7ps5DnOtCp5KjJu/lUKN2bCiUUGKJKHc8wYgUKienHOoL00LIbsLvCMQjcjoW9U+u88ZEqqvyE/qhSmTJSONdnEicbHYzy9pjJ0+RcrFyP"

/user/password

Change user password.

POST/user/password

Base URLAuthorization Token

password *

username

• rights: none, admin if setting on behalf of another user.
• verbs: POST

Parameter	Type	Required
password	String	Yes. New password.
username	String	No. Specify user if admin is changing another user's password.

curl

curl https://test-m1.minusonedb.com/user/password \
-d "password=newpassphrase" \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 user/password -password "newpassphrase"

No response when successful.

Users and Tokens

/user/list

List all users and associated metadata.

GET/user/list

Base URLAuthorization Token

• rights: admin
• verbs: GET
• parameters: none

curl

curl https://test-m1.minusonedb.com/user/list \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 user/list

▶200 OKList of users

[
  {
    "username": "user",
    "rights": [
      "schema",
      "get",
      "publish"
    ]
  },
  {
    "username": "admin",
    "rights": [
      "schema",
      "sessionQuery",
      "get",
      "publish",
      "admin",
      "sessionUpdate"
    ]
  }
]

/user/get

Retrieve metadata for specified user.

GET/user/get

Base URLAuthorization Token

username *

• rights: admin
• verbs: GET

Parameter	Type	Required
username	String	Yes

curl

curl https://test-m1.minusonedb.com/user/get?username=user \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 user/get -username user

▶200 OKUser metadata

[
  {
    "username": "user",
    "rights": [
      "schema",
      "get",
      "publish"
    ]
  }
]

/user/add

Add new user and specify rights.

POST/user/add

Base URLAuthorization Token

username *

password *

rights *

• rights: admin
• verbs: POST

Parameter	Type	Required
username	String	Yes. Must be unique.
password	String	Yes
rights	Array	Yes. List of rights to grant to user.

curl

curl https://test-m1.minusonedb.com/user/add \
-d 'username=newUser&password=passphrase&rights=schema,get' \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 user/add -username newUser \
-password passphrase -rights '["schema","get"]'

No response when successful.

/user/update

Update user metadata.

POST/user/update

Base URLAuthorization Token

username *

rights *

• rights: admin
• verbs: POST

Parameter	Type	Required
username	String	Yes
rights	Array	Yes. List of rights to grant to user.

curl

curl https://test-m1.minusonedb.com/user/update \
-d 'username=user&rights=get,publish' \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 user/update -username user -rights '["get", "publish"]'

No response when successful.

/user/remove

Remove user; the user will no longer be able to access the associated environment.

POST/user/remove

Base URLAuthorization Token

username *

• rights: admin
• verbs: POST

Parameter	Type	Required
username	String	Yes

curl

curl https://test-m1.minusonedb.com/user/remove \
-d "username=user" \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 user/remove -username user

No response when successful.

/token/list

List all tokens.

GET/token/list

Base URLAuthorization Token

• rights: admin
• verbs: GET
• parameters: none

curl

curl https://ops.minusonedb.com/token/list \
-H "m1-auth-token: $myToken"

m1 client

m1 ops token/list

▶200 OKList of active tokens

[
  {
    "expires": 1637190931886,
    "id": "f4946f10-c6d6-4345-9a8f-de6aec884186",
    "ip": "120.1.88.123",
    "username": "[email protected]"
  }
]

/token/data

Decodes the elements that make up a token.

POST/token/data

Base URLAuthorization Token

token *

• rights: admin
• verbs: POST

Parameter	Type	Required
token	String	Yes

curl

curl https://test-m1.minusonedb.com/token/data \
-d "token=eFRxichv7cTeqks99XD+iNBqR3LlgILIZ3nhKieS8c8qajV9+JcNqSqZfngvS+gJM3IBKnvI2Crd66WOIswtlr7ps5DnOtCp5KjJu/lUKN2bCiUUGKJKHc8wYgUKienHOoL00LIbsLvCMQjcjoW9U+u88ZEqqvyE/qhSmTJSONdnEicbHYzy9pjJ0+RcrFyP" \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 token/data -token eFRxichv7cTeqks99XD+iNBqR3LlgILIZ3nhKieS8c8qajV9+JcNqSqZfngvS+gJM3IBKnvI2Crd66WOIswtlr7ps5DnOtCp5KjJu/lUKN2bCiUUGKJKHc8wYgUKienHOoL00LIbsLvCMQjcjoW9U+u88ZEqqvyE/qhSmTJSONdnEicbHYzy9pjJ0+RcrFyP

▶200 OKDecoded token data

{
  "expires": 1637456697412,
  "ip": "96.253.115.241",
  "id": "687569bd-dfde-463f-b85d-dd8b1a7ea50c",
  "user": "delete"
}

/token/delete

Deletes all active tokens in an environment. This will require that all users reauthenticate.

POST/token/delete

Base URLAuthorization Token

• rights: admin
• verbs: POST
• parameters: none

curl

curl -X POST https://test-m1.minusonedb.com/token/delete \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 token/delete

No response when successful.

Schema

/schema

Retrieves current schema.

GET/schema

Base URLAuthorization Token

• rights: schema, get, publish
• verbs: GET
• parameters: none
• returns: JSON [{},...] representing the list of properties that make up the environment schema.

curl

curl https://test-m1.minusonedb.com/schema \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 schema

▶200 OKEnvironment schema

{
  "_m1rand": {
    "large": false,
    "name": "_m1rand",
    "description": "Random number assigned at publish time used for sampling use cases.",
    "type": "double",
    "required": true,
    "multi": false
  },
  "myTags": {
    "large": false,
    "name": "myTags",
    "description": "Represents important tags.",
    "type": "string",
    "required": false,
    "multi": true
  },
  "_m1key": {
    "large": false,
    "name": "_m1key",
    "description": "Unique identifier across an environment.",
    "type": "integer",
    "required": true,
    "multi": false
  },
  "myBoolean": {
    "large": false,
    "name": "myBoolean",
    "type": "boolean",
    "required": false
  }
}

/schema/add

Add properties to the environment schema. Can be called as many times as desired to add new property definitions.

POST/schema/add

Base URLAuthorization Token

properties *

• rights: schema
• verbs: POST

Parameter	Type	Required
properties	property[]	Yes

Each property is a map with the following keys:

Key	Type	Required
name	String	Yes
type	String	Yes. One of: string, text, integer, double, date.
multi	boolean	No. Default false. If true the property can contain multiple values.
description	String	No. Default null. Comment describing property, purely for informational purposes.

curl

curl https://test-m1.minusonedb.com/schema/add \
-d 'properties=[
    {
        "name": "myTags",
        "type": "string",
        "multi": true,
        "description": "Represents important tags."
    },
    {
        "name": "myBoolean",
        "type": "boolean"
    }
]' \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 schema/add -properties '[
    {
        "name": "myTags",
        "type": "string",
        "multi": true,
        "description": "Represents important tags."
    },
    {
        "name": "myBoolean",
        "type": "boolean"
    }
]'

# Or load from file:
m1 test-m1 schema/add -properties @fileWithSchema.json

No response when successful.

/schema/wipe

Deletes all user defined properties in the environment schema. Note that this does not modify data that has already been written to the data lake.

POST/schema/wipe

Base URLAuthorization Token

• rights: schema
• verbs: POST
• parameters: none

curl

curl -X POST https://test-m1.minusonedb.com/schema/wipe \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 schema/wipe

No response when successful.

Archive

Please note that the only data access to data stored in the Archive layer is via s3 file access, typically by specifying files as the file for a call to publish.

/write

Save data to the archive.

POST/write

Base URLAuthorization Token

items *

publish

async

rules

• rights: publish
• verbs: POST

Parameter	Type	Required
items	JSON map[]	Yes. Each item must be a JSON map that can contain arbitrary keys and structure.
publish	boolean	No. When true, will attempt to publish data to the data lake after archiving it.
async	boolean	No. Default true. Set to false to receive synchronous error messages in a debugging context.
rules	TransformRuleSet	No. Used only when publish is true.

curl

curl https://test-m1.minusonedb.com/write \
-d 'items=[{"property1" : {"ip" : "127.0.0.1"}, "session" : null, "event": {"id": "key98765", "time": 946702800, "type": "sessionStart"}}]&publish=true' \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 write \
-items '[{"property1":{"ip":"127.0.0.1"},"session":null,"event":{"id":"key98765","time":946702800,"type":"sessionStart"}}]' \
-publish true

No response when successful.

In a production setting, if you are publishing data immediately after archiving, we recommend you set publish-permissive to true.

Each archived item will have the origin ip address and reception time appended to it. See property-ip and property-received system parameter documentation for more details.

When geo is enabled, geographic metadata will be appended to each item based on the available ip address.

Data Lake

/publish

Load data from files in S3 or GCS into the data lake.

POST/publish

Base URLAuthorization Token

file *

rules

format

compression

encoding

delimiter

headers

rows

• rights: publish
• verbs: POST

Parameter	Type	Required
file	String	Yes. Fully qualified file path, e.g. s3://yourbucket/path/to/file.json or gs://yourgcsbucket/path/to/file.json.
rules	TransformRuleSet	No
format	String	No. One of: txt, csv, json, jsonl. If not present, filename will be inspected for format.
compression	String	No. One of: gz, zip. If not present, filename will be inspected.
encoding	String	No. Any Java encoding format. Default UTF-8.
delimiter	char	No. Character for delimited file formats. Tabs and commas assumed for txt/csv unless specified.
headers	boolean	No. Whether headers are present for delimited formats. Default true. If false, a TransformRuleSet is required.
rows	Integer	No. Max rows to load. Negative or omitted loads all rows.

curl

curl https://test-m1.minusonedb.com/publish \
-d "file=s3://m1-public/reddit/us.jsonl.gz" \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 publish -file "file=s3://m1-public/reddit/us.jsonl.gz"

▶200 OKPublish progress

"Saved 30000-46570 (16570 total)
Saved 50000-66376 (32946 total)
Saved 70000-86492 (49438 total)
Saved 90000-92104 (51542 total)"

To load data via /publish you must configure permissions so that the EC2 instance profile role associated with your environment can read the file(s) you are attempting to /publish.

// Example ReadAccess policy for S3 Configuration
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::248899197673:role/$instanceProfileRole"
      },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::$yourBucket",
        "arn:aws:s3:::$yourBucket/*"
      ]
    }
  ]
}

Configuration Steps When Publishing from S3

1. Run /env/bucket/register to register the bucket with your environment

2. In your AWS account, configure a bucket policy so that the instance profile role of your environment has read access to your data. You can obtain the instanceProfileRole from /env/get

3. Validate your bucket access by calling /publish with one of your files and rows=0. If you get an AccessDeniedException, recheck your IAM permissions above.

4. You can now call /publish on all the files you wish to load into your environment.

# create a new service account
gcloud --configuration dev iam service-accounts create whateveryouwant --display-name="Whatever you want"
# serviceAccountEmail is the email address of the service account you created above
# yourProjectId is the project in which you created your service account
gcloud --configuration dev projects add-iam-policy-binding $yourProjectId  \
          --member="serviceAccount:$serviceAccountEmail" \
          --role="roles/storage.objectViewer"

Configuration Steps When Publishing from GCS

1. Create a service account in your google cloud account and allow it to read/download files in your bucket(s)

(You can skip the step if you already have a service account that has sufficient access to the data you wish to load in your m1db environment)

2. Create a trust relationship between the EC2 instance profile role associated with your environment and your service account

# create a trust relationship between your service account and your environment instance role
# m1dbEnvironmentInstanceProfileRole is the "instanceProfileRole" attribute in the response returned by "m1 ops env/get -env your-env"
gcloud --configuration dev iam service-accounts add-iam-policy-binding $serviceAccountEmail \
--role=roles/iam.workloadIdentityUser \
--member="principalSet://iam.googleapis.com/projects/980494489932/locations/global/workloadIdentityPools/aws-pool/attribute.aws_role/arn:aws:sts::248899197673:assumed-role/$m1dbEnvironmentInstanceProfileRole"

3. Set the gcs-service-account system property to the email address of your service account.

# Set the gcs-service-account system property to point to your service account
m1 your-env system -gcs-service-account "$yourAccountEmail"

4. Enable outbound connectivity for your environment

# Enable outbound connectivity for your environment
m1 ops env/outbound -env your-env -enable true

5. Validate your GCS bucket access by calling /publish with one of your files and rows=0. If you get an access error, recheck your configuration steps. Note that it may take a few minutes for your configuration to take effect.

6. You can now call /publish on all the files you wish to load into your environment.

/next

Retrieve the next _m1key that will be assigned to a document added to the lake (via /publish, for example).

GET/next

Base URLAuthorization Token

• rights: admin, publish
• verbs: GET
• parameters: none

curl

curl https://test-m1.minusonedb.com/next \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 next

▶200 OKNext available key

10000

/get

Retrieve any number of rows from the data lake via _m1key property.

POST/get

Base URLAuthorization Token

ids *

properties

• rights: get
• verbs: GET, POST

Parameter	Type	Required
ids	long[]	Yes. IDs of records to be retrieved.
properties	Array	No. List of properties from schema to include in records. If null, all columns are returned.

• returns: JSON [{},...]

curl

curl https://test-m1.minusonedb.com/get \
-d 'ids=[10000,20000,30000]&properties=["_m1key","session.id"]' \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 get -ids "[10000,20000,30000]" -properties '["_m1key","session.id"]'

▶200 OKRetrieved records

[
  {
    "_m1key": 10000,
    "session.id": "67397f5e-b6b1-4cb9-9b88-b6396d3e1ae4"
  },
  {
    "_m1key": 20000,
    "session.id": "76a9cad0-88f8-46e3-bbae-40256f2c9267"
  },
  {
    "_m1key": 30000,
    "session.id": "acd6540d-b8cd-470e-b6e8-703cd1a48870"
  }
]

/range

Retrieve all rows from datalake with _m1key values between start (inclusive) and end (exclusive).

POST/range

Base URLAuthorization Token

start *

end *

properties

• rights: get
• verbs: GET, POST

Parameter	Type	Required
start	long	Yes. Inclusive.
end	long	Yes. Exclusive.
properties	Array	No. List of properties from schema to include in records. If null, all columns are returned.

• returns: JSON [{},...]

curl

curl https://test-m1.minusonedb.com/range \
-d 'start=10000&end=30000&properties=["_m1key","session.id"]' \
-H "m1-auth-token: $myToken"

m1 client

m1 test-m1 range -start 10000 -end 30000

▶200 OKRecords in range

[
  {
    "_m1key": 10000,
    "session.id": "67397f5e-b6b1-4cb9-9b88-b6396d3e1ae4"
  },
  {
    "_m1key": 20000,
    "session.id": "76a9cad0-88f8-46e3-bbae-40256f2c9267"
  },
  {
    "_m1key": 30000,
    "session.id": "acd6540d-b8cd-470e-b6e8-703cd1a48870"
  }
]