Reading documents

So far we have covered how to retrieve single documents using the $get() method, and how to retrieve (or list) multiple documents taking advantage either of the parent-child relationship or the membership relationship (with $listChildren(), $getMembers(), etc). Still, applications normally need more than just retrieving a single document or a list of documents.

In this document we’ll cover common use cases and how to accomplish them with the SDK.

The query function

We’ve been using the methods $get(), $getChildren() and $getMembers() with minimal configuration. But these methods also accept an additional parameter that allows us to specify constraints for the retrieval: the query parameter.

The query parameter must be a function that returns a very particular object. To help with the creation of said object, the function will receive as its only parameter a Builder object. It has the methods required to specify the desired constraints and return the object the function is expected to return.

We’ll call this function the query function.

To make things easier, here’s the general pattern to follow when using this function:

// ...
carbonldp.documents.$get( "parent/", _ => _
    .someConstraint()
    .someOtherConstraint()
);
// ...
// ...
carbonldp.documents.$get<Something>( "parent/", _ => _
    .someConstraint()
    .someOtherConstraint()
);
// ...
// ...
carbonldp.documents.$get( "parent/", function( _ ) {
    return _
        .someConstraint()
        .someOtherConstraint()
    ;
} );
// ...

Retrieving only specific properties

It is fairly common for applications to require only certain properties of a document. This can happen, for example, when rendering a list of objects and just displaying their name.

To retrieve only specific properties of a document, the Builder object provides the method properties().

Example

Let’s imagine that an application needs to render a list of employees using only their name and their profilePicture. This can be achieved doing the following:

// ...
carbonldp.documents.$getMembers( "employees/", _ => _
    .properties( {
        "name": {
            "@type": "string"
        },
        "profilePicture": {
            "@type": "string"
        }
    } )
);
// ...
// ...
carbonldp.documents.$getMembers<Employee>( "employees/", _ => _
    .properties( {
        "name": {
            "@type": "string"
        },
        "profilePicture": {
            "@type": "string"
        }
    } )
);
// ...
// ...
carbonldp.documents.$getMembers( "employees/", function( _ ) {
    return _
        .properties( {
            "name": {
                "@type": "string"
            },
            "profilePicture": {
                "@type": "string"
            }
        } )
    ;
} );
// ...

As you can see, the properties() method receives an object that looks very similar to the one used by the Object schema. The properties of this object define the property names to retrieve. Each property can specify the same things as in the Object schema, @id, @type, etc..

Now you may be wondering:

“Well, I already specified how I expect certain properties to be interpreted in my Object schema. Do I really need to declare them again?”

Of course not. If the properties were specified in the general Object schema, then you can point them to _.inherit:

.properties( {
    "name": _.inherit,
    "profilePicture": _.inherit
} )

_.inherit specifies that the property’s details should be pulled from the Object schema.

“But what if I don’t want to declare my properties on the general object schema?”

Like we said in the Object schema document, instead of declaring them in the general object schema it is a good practice to declare properties in the object schema of a specific type. But if you do that, the Builder object won’t have information to use for the properties specified.

We will address this problem in the next section.

When you query the properties of a document, by default, Carbon looks in the values for a match. When you use ._inherit, Carbon states the property as optional in the query, which means that it will try to look for the values inside that property, but if it doesn’t find them, it will still return all other properties stated in the query and the reference to the document. If you want to specify that you want a certain property to be required in the results, instead of _.inherit, you may use {required: true}. This tells Carbon to look for documents that have that property.

If Carbon finds a document that doesn’t have that property, it will just ignore it and won’t return it as a result of the query, even tough it has other properties. All required properties must exist in the document for it to be returned.

properties( {
    "name": {
        required: true // requires the name property
    },
    "profilePicture": _.inherit // the profile picture is optional
} )

Note: If the $get() can’t find any document matching the required parameter, it will return a 404 NOT FOUND error. However, the $getMembers() method will return an empty array.

Retrieving only documents of a specific type

If you have not worked with object schemas before or have no idea of what they are, please refer to the What is an object schema? section, particularly the Data Types (@type) section, to better understand the use of types while querying.

 

A document can store any type of documents as its children/members, but most of the time, applications only care about documents that follow a particular shape. Or said in other words, have a specific @type.

The Builder object provides the method withType() to specify only documents that have a desired @type that should be retrieved.

// ...
carbonldp.documents.$getMembers( "employees/", _ => _
    .withType( "Employee" )
);
// ...
// ...
carbonldp.documents.$getMembers<Employee>( "employees/", _ => _
    .withType( "Employee" )
);
// ...
// ...
carbonldp.documents.$getMembers( "employees/", function( _ ) {
    return _
        .withType( "Employee" )
    ;
} );
// ...

Filtering documents based on their type has an additional advantage, it provides additional information about the desired documents to the Builder object and because of this, the _.inherit object can now be used with properties that are located in that @type‘s schema:

// ...
carbonldp.extendObjectSchema( "Employee", {
    "name": {
        "@type": "string"
    },
    "profilePicture": {
        "@type": "string"
    }
} );

carbonldp.documents.$getMembers( "employees/", _ => _
    .withType( "Employee" )
    .properties( {
        "name": _.inherit,
        "profilePicture": _.inherit
    } )
).then( ( employees ) => {
    console.log( employees );
} );
// ...
// ...
carbonldp.extendObjectSchema( "Employee", {
    "name": {
        "@type": "string"
    },
    "profilePicture": {
        "@type": "string"
    }
} );

carbonldp.documents.$getMembers<Employee>( "employees/", _ => _
    .withType( "Employee" )
    .properties( {
        "name": _.inherit,
        "profilePicture": _.inherit
    } )
).then( ( employees:(Employee & Document)[] ) => {
    console.log( employees );
} );
// ...
// ...
carbonldp.extendObjectSchema( "Employee", {
    "name": {
        "@type": "string"
    },
    "profilePicture": {
        "@type": "string"
    }
} );

carbonldp.documents.$getMembers( "employees/", function( _ ) {
    return _
        .withType( "Employee" )
        .properties( {
            "name": _.inherit,
            "profilePicture": _.inherit
        } )
    ;
} ).then( function( employees ) {
    console.log( employees );
} );
// ...

Filtering documents

The method withType() can filter documents based on their types, but it is fairly common for applications to need to filter them based on their properties instead (or more complex conditions).

The Builder object provides the method filter() which can be used to filter the documents to be retrieved based on a custom condition.

This method accepts a SPARQL FILTER expression (you can find more about them here). If you don’t
know SPARQL, don’t worry. The Builder object also provides methods to create those expressions without SPARQL knowledge.

Alongside with .filter(), Carbon offers the .values() method. This method adds a filter to the specific values of the property where the query is being applied.

Using the Object() method

The .object() method works similarly to the .value() method, but instead of working with literals, it works with documents. You provide the $id of the object and this method will return a reference to that document in a format which the builder can understand.

    __.object(post1.$id)
    __.object("employees/john-snow/") // works too!

Using the values() method

Unlike filter(), it works by giving it a value (or document) which have to exactly match to the property in the query document. (It is also case sensitive!).

You can use the previous tools with values() too, since it can receive all the parameters you need.

    query: __ => __
    .values(    // Make sure to only bring values...
        __.object( // ...that match with the following id...
            post1.$id;
        ),
        __.value( // ...and the following name...
            "John Snow"
        ),
        __.value( // ...and this age
            33
        ),
        /* etc */
    ),

Comparison between .filter() and .values()

.filter() is much more versatile as it allows you to use some of SPARQL‘s power directly. .values() can only do exact matches.

    .filter(`${_.property('myProperty')} = "Some value"`)).
    // Is the same as..
    .values(_.value("Some value"))

Note: In the current version of Carbon LDP™, .filter() is not compatible with .limit() and .offset(). While your code will compile and run, you query will not yield the results you would normally expect. On the other hand, .values() does work as expected with the aforementioned utilities. This means that a paginated query is not attainable with .filter(). while it is possible with .values().

Comparison operators

Filter expressions accept the following comparison operators:

= Equal to
!= Not equal to
> More than
< Less than
>= More than or equal to
<= Less than or equal to

Using JavaScript values

Values in filter expressions need to be wrapped using the _.value() method. This makes the Builder object convert the native JavaScript value to a SPARQL value. Example:

_.value( "Hello!" )
_.value( 1289.9 )
_.value( someVariable )
_.value( new Date() )
// etc

Referring to properties

To reference a property of the document in the filter expression, the Builder object provides the method: _.property(). It accepts the name of the property to reference:

_.property( "name" )

Comparing values

Using these three tools comparison expressions can be used in the filter expression. For example, if we wanted to get only blog posts that had a publishedOn value previous to the current date (meaning they are already published), we could accomplish it using the following code:

// ...
carbonldp.documents.$getMembers( "posts/", _ => _
    .properties( {
        "title": {
            "@type": "string"
        },
        "content": {
            "@type": "string"
        },
        "publishedOn": {
            "@type": "dateTime"
        }
    } )
    .filter( `${_.property( "publishedOn" )} < ${_.value( new Date() )}` )
);
// ...
// ...
carbonldp.documents.$getMembers<BlogPost>( "posts/", _ => _
    .properties( {
        "title": {
            "@type": "string"
        },
        "content": {
            "@type": "string"
        },
        "publishedOn": {
            "@type": "dateTime"
        }
    } )
    .filter( `${_.property( "publishedOn" )} < ${_.value( new Date() )}` )
);
// ...
// ...
carbonldp.documents.$getMembers( "posts/", function( _ ) {
    return _
        .properties( {
            "title": {
                "@type": "string"
            },
            "content": {
                "@type": "string"
            },
            "publishedOn": {
                "@type": "dateTime"
            }
        } )
        .filter( _.property( "publishedOn" ) + " < " + _.value( new Date() ) )
    ;
} );
// ...

Parenthesis and logical operators

Filter expressions accept parenthesis to wrap comparisons or arithmetic expressions. They can be used together with the following supported logical operators:

&& And
|| Or

For example, to retrieve published blog posts that don’t have the property deleted set to true, the following code could be used:

// ...
carbonldp.documents.$getMembers( "posts/", _ => _
    .properties( {
        "title": {
            "@type": "string"
        },
        "content": {
            "@type": "string"
        },
        "publishedOn": {
            "@type": "dateTime"
        },
        "deleted": {
            "@type": "boolean"
        }
    } )
    .filter( `( ${_.property( "publishedOn" )} < ${_.value( new Date() )} ) && ( ${_.property( "deleted" )} != ${_.value( true )} )` )
);
// ...
// ...
carbonldp.documents.$getMembers<BlogPost>( "posts/", _ => _
    .properties( {
        "title": {
            "@type": "string"
        },
        "content": {
            "@type": "string"
        },
        "publishedOn": {
            "@type": "dateTime"
        },
        "deleted": {
            "@type": "boolean"
        }
    } )
    .filter( `( ${_.property( "publishedOn" )} < ${_.value( new Date() )} ) && ( ${_.property( "deleted" )} != ${_.value( true )} )` )
);
// ...
// ...
carbonldp.documents.$getMembers( "posts/", function( _ ) {
    return _
        .properties( {
            "title": {
                "@type": "string"
            },
            "content": {
                "@type": "string"
            },
            "publishedOn": {
                "@type": "dateTime"
            },
            "deleted": {
                "@type": "boolean"
            }
        } )
        .filter( "( " + _.property( "publishedOn" ) + " < " + _.value( new Date() ) + " ) && ( " + _.property( "deleted" ) + " != " + _.value( true ) + " )"  )
    ;
} );
// ...

Retrieving all properties from a document

Sometimes, you may need to access all the properties from a certain document. To do this, you can use _.all inside the .properties() method within your query.

carbonldp.documents.$getMembers( "employees/", _ => _
    .properties(_.all) //returns an array of employees with all of their properties.
);

Tip: _.all doesn’t bring the properties from fragments within a document, just references to these fragments. However you can nest .properties() to get them:

await this.carbon.documents.$getMembers("employees/", _ => _
    .properties(_.all)
    .properties({contactInfo: { //contactInfo is the name of the fragment
        query: _ => _
            .properties(_.all)
    }})
)

Remember that since fragments are like documents, they are more expensive to query. The type of query mentioned above is recommended only when you need to bring properties from a fragment and the document. If you just want the properties from the document, a simple $get() should be enough.

Paging through ordered documents

Sometimes collections can get very big and retrieving them all together on a single request can take a long time (or even crash). To avoid this, the Builder object provides three methods that let you page through the documents of a collection:

orderBy()

Orders the documents to be retrieved based on one of their properties. This method accepts two parameters:

  • The property name
  • Either "ascending"/"ASC" or "descending"/"DESC" to specify the order
limit() Number of documents to retrieve
offset() Indicates the number of documents to ignore when retrieving results. An offset of 0 has no effect

Note: Using limit() and offset() without orderBy() can have unpredictable results as the order will be decided by the platform (or there may be no order at all).

Example

Continuing with the Blog example, let’s imagine that we needed to retrieve the 10 most recent posts for the post feed. This could be achieved with the following code:

// ...
carbonldp.documents.$getMembers( "posts/", _ => _
    .properties( {
        "title": {
            "@type": "string"
        },
        "content": {
            "@type": "string"
        },
        "publishedOn": {
            "@type": "dateTime"
        },
        "deleted": {
            "@type": "boolean"
        }
    } )
    .filter( `( ${_.property( "publishedOn" )} < ${_.value( new Date() )} ) && ( ${_.property( "deleted" )} != ${_.value( true )} )` )
    .orderBy( "publishedOn", "descending" )
    .limit( 10 )
);
// ...
// ...
carbonldp.documents.$getMembers<BlogPost>( "posts/", _ => _
    .properties( {
        "title": {
            "@type": "string"
        },
        "content": {
            "@type": "string"
        },
        "publishedOn": {
            "@type": "dateTime"
        },
        "deleted": {
            "@type": "boolean"
        }
    } )
    .filter( `( ${_.property( "publishedOn" )} < ${_.value( new Date() )} ) && ( ${_.property( "deleted" )} != ${_.value( true )} )` )
    .orderBy( "publishedOn", "descending" )
    .limit( 10 )
);
// ...
// ...
carbonldp.documents.$getMembers( "posts/", function( _ ) {
    return _
        .properties( {
            "title": {
                "@type": "string"
            },
            "content": {
                "@type": "string"
            },
            "publishedOn": {
                "@type": "dateTime"
            },
            "deleted": {
                "@type": "boolean"
            }
        } )
        .filter( "( " + _.property( "publishedOn" ) + " < " + _.value( new Date() ) + " ) && ( " + _.property( "deleted" ) + " != " + _.value( true ) + " )"  )
        .orderBy( "publishedOn", "descending" )
        .limit( 10 )
    ;
} );
// ...
After that, if the applicat

After that, if the application needed to retrieve the next 10 posts the only change needed would be to add the offset() method:

// ...
.orderBy( "publishedOn", "descending" )
.limit( 10 )
.offset( 10 )
// ...

Retrieving nested objects

So far we have covered only simple object structures. But sometimes an application needs to retrieve a more complex one, like for example the following objects tree:

{
    "$id": "employees/john-snow/",
    "types": [ "Employee" ],
    "name": "John Snow",
    "profilePicture": "http://.../profile.png",
    "projects": [
        {
            "$id": "projects/website-redesign/",
            "types": [ "Project" ],
            "title": "Website redesign"
        },
        {
            "$id": "projects/acme-intranet/",
            "types": [ "Project" ],
            "title": "ACME's Intranet"
        }
    ]
}

Retrieving connected objects can be accomplished by specifying that the value of a property (of @id type) should be included in the response. To do it, a query property needs to be declared in a property description object.

This query property accepts another query function that will be used to apply constraints to the values of that
property.

In this query function the desired properties of the nested object can be specified too (along with everything we have included covered so far).

To retrieve the previous objects tree the following code could be used:

// ...
carbonldp.extendObjectSchema( "Employee", {
    "name": {
        "@type": "string"
    },
    "profilePicture": {
        "@type": "string"
    },
    "projects": {
        "@type": "@id"
    }
} );
carbonldp.extendObjectSchema( "Project", {
    "title": {
        "@type": "string"
    }
} );

carbonldp.documents.$get( "employees/john-snow/", _ => _
    .withType( "Employee" )
    .properties( {
        "name": _.inherit,
        "profilePicture": _.inherit,
        "projects": {
            "query": _ => _
                .withType( "Project" )
                .properties( {
                    "title": _.inherit
                } )
        }
    } )
).then( ( employee ) => {
    console.log( employee.projects.length ); // 2
    console.log( employee.projects[ 0 ].title ); // Website redesign
} );
// ...
// ...
carbonldp.extendObjectSchema( "Employee", {
    "name": {
        "@type": "string"
    },
    "profilePicture": {
        "@type": "string"
    },
    "projects": {
        "@type": "@id"
    }
} );
carbonldp.extendObjectSchema( "Project", {
    "title": {
        "@type": "string"
    }
} );

carbonldp.documents.$get<Employee>( "employees/john-snow/", _ => _
    .withType( "Employee" )
    .properties( {
        "name": _.inherit,
        "profilePicture": _.inherit,
        "projects": {
            "query": _ => _
                .withType( "Project" )
                .properties( {
                    "title": _.inherit
                } )
        }
    } )
).then( ( employee:Employee & Document ) => {
    console.log( employee.projects.length ); // 2
    console.log( employee.projects[ 0 ].title ); // Website redesign
} );
// ...
// ...
carbonldp.extendObjectSchema( "Employee", {
    "name": {
        "@type": "string"
    },
    "profilePicture": {
        "@type": "string"
    },
    "projects": {
        "@type": "@id"
    }
} );

carbonldp.documents.$get( "employees/john-snow/", function( _ ) {
    return _
        .withType( "Employee" )
        .properties( {
            "name": _.inherit,
            "profilePicture": _.inherit,
            "projects": {
                "query": function( _ ) {
                    return _
                        .withType( "Project" )
                        .properties( {
                            "title": _.inherit
                        } )
                    ;
                }
            }
        } )
    ;
} ).then( function( employee ) {
    console.log( employee.projects.length ); // 2
    console.log( employee.projects[ 0 ].title ); // Website redesign
} );
// ...

Even though the projects property isn’t pointing to _.inherit, it will still inherit whatever wasn’t specified on the description object from the object schema. In this case, the @type gets inherited since it wasn’t explicitly defined.

Note that nested queries can be as deep as you need them to be but the requests may become slow.

Virtual properties

Apart from querying the properties a document has, you may also query virtual properties. These are properties that don’t actually exist inside the document, but are queried from other documents. To achieve this, you must use property paths.

    let queryResults = await this.carbon.documents.$getMembers("employees/", _ => _
        .properties({
            name: _.inherit,
            nameOfFamilyMembers: {
                path: __ =>
                    __.sequences(EMPLOYEE_SCHEMA.familyMembers["@id"], PERSON_SHEMA.name["@id"])
            }
        })
    )

Conclusion

By specifying a query function, the application can control the data it is going to retrieve and reduce the number of requests it executes.

But sometimes these capabilities aren’t enough. After all, the query function isn’t as flexible as a fully-fledged query language.

In the next document we’ll cover how to use SPARQL, a query language that can cover the use cases the query function can’t.