Querying with Vyne
Submitting queries to Vyne's API to discover data
Vyne focuses on querying for data based on it's meaning, rather than which system provides it. This allows services to change, and data to move, without requiring consumers to update their queries.
Writing queries
Queries are written in TaxiQL, an open source query language for data.
TaxiQL is a great query language.
The taxi documentation has details on the syntax, which we haven't duplicated here. Go check it out, then come back.
We'll wait.
TaxiQL is agnostic of where data comes from - it's left to Vyne to discover data from the various sources that have been connected.
Here's some sample queries:
// Find all the movies
find { Movie[] }
// Find all the movies, enriching and projecting them to a different structure
find { Movie[] } as {
title : MovieTitle
director : DirectorName
rating : RottenTomatoesScore
}[]
Projections
Projections are a way of taking data from one place, then transforming & combining it with other data sources.
Vyne uses the information present on the object being projected in order to call services and find other information.
eg:
model Purchase {
transactionId : TransactionId
customerId : CustomerId
}
find { Purchases[] }
as {
// Projections let you change field names, and reshape objects as required
txn: TransactionId
// Not present on the original Purchase object, so try to
// find it using something we already know (in this case, the CustomerId)
customerName: CustomerName
}
Data discovery rules
When projecting, Vyne will use information present on the source object to discover data on the target object.
Data can be fetched from a single operation that returns the value, or by invoking a chain of operations to return the value.
Operations with @Id fields on return types
If the result of an operation is an object that exposes an @Id
field, then only operations which accept that @Id
field as
an input will be called.
eg:
model Customer {
@Id customerId : CustomerId
name : CustomerName
}
service CustomerService {
// Can be called when projecting, because
// Person has an @Id of type PersonId
findCustomer(CustomerId):Customer
// Cannot be called when projecting, because
// Person has an @Id, and it isn't PersonName
findCustomerByName(CustomerName):Customer
}
Operations without @Id fields on return types
If the result of an operation is an object that does not expose an @Id
field, then it can be called with
any information available.
Filling in nulls
By default, if a service returns a null value, Vyne will accept it as-is.
However, if query annotates a field on a projection type with @FirstNotEmpty
, Vyne will
attempt to populate values by invoking operations to populate the missing values.
Vyne will execute a search using the other values present on the entity being projected as potential inputs to operations, and build a path to populate the missing values.
Operations are invoked following the standard Data Discovery Rules
Understanding caching in Vyne
Vyne does not maintain a long-lived cache between operations, though this is planned for a future release.
However, Vyne will cache operations for the lifetime of a query. This prevents the same operation being invoked repeatedly while projecting multiple rows in a result.
Responses are cached for a given operation + set of inputs. If an operation is invoked with different parameters, the cache is not used.
Operations that return an array of results, which return more than 10 values, will not have their responses cached. (This is not currently configurable, but reach out on slack if you need to configure this).
Recovering from failure
If an operation returns an error while Vyne is attempting to execute a query, then it is excluded from being invoked with the same parameters again. This exclusion is scoped to the query only, and expires at the end of the query.
After excluding the operation, Vyne will attempt to find another path to return the value being discovered.
Expressions in queries
Taxi allows the definition of expressions on both types and fields, but doesn't provide an evaluation engine - that's where Vyne comes in.
Typically, expressions are used in a projection within a query.
You can also use them on a model to expose derived information when a model is parsed by Vyne (eg., when return from a service) - but that's less common. So, while documentation here focuses on query projections, you can do everything here on a model too.
Writing an expression in a projection
Expressions can be defined in the fields of a projected result from a query:
find { Flights[] }
as {
flightNumber : FlightNumber
totalSeatsAvailable : TotalSeats
soldSeats : SoldSeats
remainingSeats : Int by (this.totalSeatsAvailable - this.soldSeats)
}
Expressions can be defined in two ways - on a field, or on a type.
Expressions on a field
// Expression types on a field:
find { Flights[] }
as {
flightNumber : FlightNumber
totalSeatsAvailable : TotalSeats
soldSeats : SoldSeats
// field expressions can be defined EITHER using field references...
remainingSeats : Int by (this.totalSeatsAvailable - this.soldSeats)
// ...or type references...
remainingSeats : Int by (TotalSeats - SoldSeats)
}
Expressions on a type
To encapsulate common expressions, you can define a type with the expression:
// Expression type:
type RemainingSeats by TotalSeats - SoldSeats
// Which is then used on a projection:
find { Flights[] }
as {
flightNumber : FlightNumber
totalSeatsAvailable : TotalSeats
soldSeats : SoldSeats
remainingSeats : RemainingSeats
}
Unlike field expressions, type expression cannot use field names, and can only reference other types.
How Vyne discovers values to evaluate expressions
When Vyne is evaluating an expression, it first looks on the source object being projected for the input values into the expression.
If any inputs are not available, then Vyne will perform a search using the current data available on the source object in an attempt to look up the value.
Submitting queries
Generally, developers will use the UI to write and test their queries, then integrate using Vyne's rest API.
Rest API
Queries to Vyne are submitted to the /api/taxiql
endpoint:
curl 'https://localhost:9022/api/taxiql' \
-H 'Content-Type: application/taxiql' \
--data-raw 'find { Movie[] }'
A word about content type
Strictly speaking, the content type for taxiql queries is application/taxiql
. However, the Vyne server will accept
taxiql queries with any of the following content types headers:
Content-Type: application/json
Content-Type: application/taxiql
Content-Type: text/plain
This is to allow broad compatability with clients.
Large queries with Server Sent Events
Running large queries can result in out-of-memory errors if Vyne is holding the result set in memory.
To address this, Vyne supports pushing results over server-sent-events. To consume a query as a server-sent-event, set
the Accept
header to text/event-stream
:
curl 'http://localhost:9022/api/taxiql' \
-H 'Accept: text/event-stream' \
-H 'Content-Type: application/taxiql' \
--data-raw 'find { Movie[] }'
Results are pushed out from Vyne as they are available.
Including type metadata in responses
Vyne can include type metadata in the responses being sent back.
To enable this, append ?resultMode=TYPED
to the API call:
curl 'http://localhost:9022/api/taxiql?resultMode=TYPED' \
-H 'Accept: text/event-stream' \
-H 'Content-Type: application/taxiql' \
--data-raw 'find { Movie[] }'
Defining output formats
By default, Vyne serves results to queries as JSON.
This can be configured to customize the result format.
With Accept headers
The following accept headers are supported:
Header | Result type |
---|---|
application/json | json |
application/csv | csv |
`text/event-stream | JSON with server-sent-events |
Defining output formats with model formats
Fine-grained control is supported with custom model specs defined on model types. At present, only limited support is provided, but we plan to provide additional formats in a future release, along with the ability to register bespoke formats.
Formats are defined by adding an annotation to the model defined as the output type.
For example:
import io.vyne.formats.Csv
@Csv(
delimiter = "|",
nullValue = "NULL"
)
model Person {
firstName : String by column("firstName")
lastName : String by column("lastName")
age : Int by column("age")
}
// Query:
// Response type (Person) contains a Csv format defined,
// which will be considered when writing responses.
find { Customer[] }
as { Person[] }
Csv
The full definition of the Csv model format is as follows:
Parameter | Description | Required | Default Value |
---|---|---|---|
delimiter | Defines the delimiter to use between columns | false | , |
firstRecordAsHeader | Indicates if the first line should be treated as a header | false | true |
nullValue | Defines a custom token to use in place of null | false | null |
containsTrailingDelimiters | Indicates if the last delimiter is an empty column which should be ignored | false | false |
withQuote | Defines a quote character used if content needs to be escaped | false | " |