Legacy
release-2020-12
December 2020 Release
https://github.com/livingdocsIO/livingdocs-editor/tree/release-2020-12

Attention: If you skipped one or more releases, please also check the release-notes of the skipped ones.

Newsletter

Webinar

Features

  • Recording: here (code: 6Tvm^5U8)
  • Documentation: here

Developers

  • Recording: here (code: #0jfi47c)
  • Slides: here

System Requirements

Minimal

  • Elasticsearch 6.8.5 🔥

Suggested

  • Elasticsearch 7
  • Base Docker Images
    • livingdocs-server: livingdocs/server-base:14.3
    • livingdocs-editor: livingdocs/editor-base:14.3

Highlights

When selecting text in the editor, the link tool in the editable toolbar allows to link to documents instead of URLs only. The input field is a combined input and detects if you want to search, or if you entered an URL.

Attention: The delivery needs a strategy to redirect id route patterns in order for inline links to work in the delivery.

image

  • If you search, you get back a list of internal documents
  • If you paste an URL that matches one of your deliveries, the link is automatically upgraded to a document reference.

References:

Custom Indexes / Queue Base Work 🎉

Queue Base Work

To support all kind of features which need (job-)queues (now/future), we did some base work. These are the most important changes:

Custom Indexes

Some customers want to have their customised Elasticsearch publication index to make customised search requests. The switch to Redis streams was necessary to make custom indexes possible. Why you could need a custom index and how to setup one, can be found here.

Migration Guide If you already have implemented a custom index in your downstream and want to replace it with the Livingdocs custom index solution, please contact us to make a planning. The upgrade is not difficult, but every customer is different and therefore it needs individual planning.

References:

Cloudinary Storage Support 🎉

Beside Amazon S3 we introduced Cloudinary as storage. Look into the Server PR for instructions.

References:

More Secure Authentication Workflow 🎉

We worked on the security for the user authentication in Livingdocs. Some improvements are:

  • Increased security for the accessTokens
  • if an accessToken is stolen, it can’t be used without a valid client cookie
  • accessTokens can’t renew themselves anymore
  • an accessToken is bound to a valid client and session

For more information, read here.

References:

Airship Integration for Push Notifications 🎉

We integrated Airship for push notifications.

References:

Experimental

Mobile - Inline Editing 🎉

With this release, we allow a user to inline add/edit components and its settings in the editor with your mobile. This is a MVP, but we will gradually improve the inline editing in the next few releases.

References:

Editable Teasers 🎉

Editable teasers are embedded editable Livingdocs components. Technically editable teasers are doc-includes returning Livingdocs components, which can be edited like any other component. For more information read the documentation and look into the example on the example-server.

Attention: Editable teasers do not work with the render pipeline v1 (which most of the customers are using at the moment). This should be fixed in an upcoming release.

References:

Videos 🎉

We introduce videos in Livingdocs with these abilities:

  • upload videos and set metadata in media library
  • upload videos and set metadata in editor via drag + drop / upload button
  • import videos via public API
  • Add project configuration for mediaVideo MediaType
  • Add new directive doc-video in a Livingdocs design

In the upcoming releases we will bring in some improvements and make the video feature more stable.

References:

Breaking Changes 🔥

Migrate the database 🔥

It’s a simple/fast migration with no expected data losses.

# run grunt migrate to update to the newest database schema
# migration - 142-add-legacy-design-name.js
#   rename document_migrations.design_name to target_design_name
# migration - 143-drop-unused-postgres-extensions.js
#   drop unused uuid-ossp extension
# migration - 144-add-missing-primary-keys.js
#   add missing primary keys to tables to support replication better
# migration - 145-document-content-types.js
#   introduce document_content_types table to support migrations better in the future
# migration - 146-add-metadata-id-to-revisions.js
#   add document_revisions.metadata_id to support metadata to document/publication relations better in the future
# migration - 147-add-user-sessions.js
#   add user_sessions table to support new auth workflow
livingdocs-server migrate up

Drop Support for Elasticsearch < 6.8.5 🔥

🔥 The support for Elasticsearch versions < 6.8.5 has been dropped. Please update your Elasticsearch cluster to Elasticsearch >= 6.8.5.

Important! You have to do an Elasticsearch update to >=6.8.5 before installing the December release. How to to a rolling upgrade is documented here.

References: Server PR

Elasticsearch Indexes 🔥

Server Configuration

  • 🔥 moved server config search.articlePublicationIndexEnabled to elasticIndex.documentPublicationIndexEnabled
  • 🔥 removed server config search.articlePublicationIndex. The publication index name is now auto generated: ${indexNamePrefix}-li-document-publications-index

CLI

  • 🔥 removed livingdocs-server es-publication-delete-index use livingdocs-server elasticsearch-delete-index --handle=li-publications -y instead
  • 🔥 removed livingdocs-server es-publication-reindex use livingdocs-server elasticsearch-index --handle=li-publications -y instead

Environment Config

We automatically create a publication index for the public API. Therefore you must define an indexNamePrefix for every environment. The definition of indexNamePrefix is free, but we suggest a pattern like ${your-company}-${environment}.

🔥 Define an indexNamePrefix for every environment

elasticIndex: {

  // every index name will be prefixed to prevent name clashes
  // the index will be created with this pattern: `${indexNamePrefix}-${handle}-index`
  indexNamePrefix: 'your-company-local',
}

Publication Index for public API

🔥 Run a background index job for the publication index

To support future updates, we did an internal update where we define Elasticsearch aliases pointing on an index. With this change, you need to re-index the publication index used for the public API search endpoint.

After the deployment, please immediately run this CLI task (the publication index will be empty after the deployment):

livingdocs-server elasticsearch-index --handle=li-publications -y

🔥 When using the public API search endpoint (/api/v1/publications/search), then you have to reindex all publications, with the command livingdocs-server elasticsearch-index --handle=li-publications -y.

When you start the new server version, the publication index is empty. To use the publication index in production for the public API as fast as possible, you have 2 options:

    1. Start a sidekick container with the new server version and make a full background index with livingdocs-server elasticsearch-index --handle=li-publications -y. As soon as it’s done, you can start your casual servers with the new server version
    1. If you want to deploy/start your new server version without a preparation step, you can index the most recent publication with an additional parameter --since. For example livingdocs-server elasticsearch-index --handle=li-publications --since 1m -y does indexing all publications published within the last month. As soon as you’re done with this indexing step, you can run another background without --since argument to index all publications.

Authentication

With the improved authentication workflow, we have some additional breaking changes.

🔥 third party applications / e2e-tests may need some adaptions correctly supporting cookies

🔥 Local development on Chrome now requires a SSL setup for authentication to work. Setting up a certificate locally and proxying requests using the editor environment config is advised.

// advised configs for local development
module.exports= {
  // https is not _required_ but there may be some complications such as cookies being filtered when trying to overwrite secure cookies or other behavior that's dependant on the browser vendor
  https: require('../cert'),
  api: {
    // disabling the proxiedHost config (CORS-mode)
    // will only work in production
    // or a server with a valid SSL setup
    proxiedHost: 'http://localhost:9090'
  }
}

🔥 all requests need to allow sending credentials or forward cookies if requests are made with user/project tokens. (API Tokens should still work exactly as before!)

🔥 (3rd-party) applications that use the /auth/local/login endpoint, need to support cookies. It should be as easy as forwarding the liClient cookie.

🔥 For security reasons CORS is now disabled by default. We encourage a more secure approach where we forward a path on the editor domain to the server instance. For Example: ’https://edit.livingdocs.io/proxy/api/’ should be forwarded to ‘https://server.livingdocs.io’

  • This guards against CSRF attacks
  • Latency improvements as requests are halved (no more OPTIONS requests)
  • Cookies are more secure due to the possibility of using the sameSite: ‘strict’ option.

API Changes

🔥 removed authApi.getLoginHistory

🔥 removed authApi.revokeAccessTokensOfUser

🔥 removed authApi.reissueAccessToken

🔥 removed authApi.revokeAccessToken

🔥 changed authApi.createAccessTokenForTests -> authApi._createAccessTokenForTests | now returns token instead of {token}

🔥 move authApi.authorizationMiddleware out of the authApi and do not expose it anymore

🔥 Removed authUtils. authenticationSuccess (The promise version is still the same)

Routes 🔥 /authenticate has moved to /auth/local/login

🔥 /users/me has moved to /auth/me

🔥 removed POST /authenticate/refresh

🔥 removed POST /token/revoke

🔥 removed POST /token/reissue -> access tokens are reissued on /auth/me now

🔥 removed POST /token/revoke-tokens

🔥 changed GET /users/:id does not return a login history anymore

SSO Logins 🔥

With the improved authentication workflow, we have some additional breaking changes for SSO Logins. If the callback URL for SSO does not match the editorUrl, we set the auth cookies for a wrong domain leading to issues when logging in.

Migrating an existing SSO login

With cookies being set on a new URL, the SSO Logins need to be re-configured. Do the following:

  1. Make sure your editor__public_host env variable is set to the editor URL (e.g. https://edit.livingdocs.io)
  2. Change all callback URLs for your SSO provider to the pattern https://{editorUrl}/proxy/auth/{provider}/callback. For the Livingdocs service this looked e.g. as follows: auth__connections__github__config__callbackURL = https://edit.livingdocs.io/proxy/auth/github/callback. (NOTE: depending on your traefik setup it might also require proxy/api instead of /proxy).
  3. In the settings for your social providers, allow the new callback URLs (for FB for example we had to allow the redirect URL https://edit.livingdocs.io/proxy/auth/facebook/callback in our Facebook app)

Migration to Redis Streams 🔥

  • 🔥 Existing messages from bull won’t be processed anymore.
  • 🔥 If you had pending index or imports, you’ll need to restart them. Our whole setup already supported retries everywhere except in the importer of the public api. It’s probably best if you re-trigger the imports after deployment.
  • ❌ Removed bull dashboard and replaced it with an operations screen in the admin interface of the editor.

References: Server PR

Slug Metadata in Slug Routing Pattern 🔥

If you activated the routing feature and if you are using a route pattern containing :slug and at the same time you have a metadata property with the handle slug the behavior will change in such a way that the route pattern for :slug will be built out of the slug metadata property value and not the document title. In most cases this is what you want. If it is not, you can rename the handle of your existing slug metadata property to prevent clashes.

References: Server PR

DocumentsApi - Removed functions 🔥

  • 🔥 Removed DocumentEntity.find and DocumentEntity.findById methods, please use the official apis
  • 🔥 Removed RevisionEntity.find and RevisionEntity.findById methods, please use the official apis
  • 🔥 Changed parameters of documentApi.getLocks from (documentId) to ({projectId, documentId}), so we can save a few roundtrips to check whether a user is allowed to access the document.

References: Server PR

S3 Storage Default 🔥

🔥 The S3 default changed from {ACL: 'public-read'} to undefined - it now uses the bucket ACL-defaults.

How to revert to the behaviour before the release 🔥

To restore the behaviour you can explicitly pass the {ACL: 'public-read'} in (images|files).storage.config.params.ACL

storage: {
  strategy: 's3',
  config: {
    bucket: 'li-bucket-dev',
    region: 'eu-central-1',
    accessKeyId: 'secret',
    secretAccessKey: 'secret',
    params: {
      ACL: 'public-read' // <--------- add this to go back to the old behavior
    }
  }
}

References: Server PR

DocumentRelationApi - Changed Params 🔥

🔥 Changed params on documentRelationApi.setDocumentEmbedReferences from (documentVersion) to ({documentVersion, trx})

NOTE: The new param trx is optional and only necessary if you want to call within a transactional context. If you do pass it, you are responsible to call trx.commit or trx.rollback when you’re done.

References: Server PR

Index Feature - Removed publicationTransformModule 🔥

  • 🔥 Remove server config option search.indexer.publicationTransformModule
  • 🔥 Remove parameter publication-transform-module in livingdocs-server es-publication-reindex task

Note: This change should have no consequences for customers.

References: Server PR

Index Feature - Removed/Renamed Functions 🔥

  • 🔥 Removed req.accessToken from all requests. Please migrate to req.verifiedToken.bearerToken
  • 🔥 Removed searchManager.putDocument. Please migrate to indexingApi.addJob that executes the same method internally

References: Server PR

Security - User access tokens require valid cookies

  • 🔥 Cypress helpers need some adaptions to correctly support cookies.

Our core implementation can serve as reference for a cypress login helper: https://github.com/livingdocsIO/livingdocs-editor/blob/master/cypress/support/liLogin.js/#L69

  • 🔥 All requests against the server using an user access token require a valid session and the cookies that belong to it.

You may need to allow credentials or forwarding cookies if you have been working with User tokens. (API Tokens should still work exactly as before!)

Example an axios instance using the withCredentials flag

const axios = require('axios').create({..., withCredentials: true})

Editor CSS Class Renamings 🔥

🔥 If you’ve done CSS modifications based on the original upstream classes in the editor please look into this PR. We did a lot of small refactorings/renamings.

References: Editor PR

CSS Stylesheets vh is not supported anymore 🔥

Because of changes in the editor we need to replace the vh unit in the design assets. This is done automatically but the css assets in the design should support CORS when the files are on a different domain then your Livingdocs Editor. Otherwise we can’t read the CSS Rules and it can lead to an unexpected behavior in the editor. If CORS can’t be enabled on the css assets it should be replaced with a unit which does not directly base on the height of the viewport.

APIs 🎁

Public API

For all endpoints documentation, look into your editor’s public API doc - ‘https://your-editor.com/public-api'.

Added Endpoint for a MediaLibrary Import 🎁

🎁 POST /api/v1/import/mediaLibrary

References:

Added Endpoint for a MediaLibrary Metadata Update 🎁

🎁 PATCH /api/v1/mediaLibrary/:id

References:

Added Endpoints for Document Lists 🎁

  • 🎁 GET /api/v1/document-lists Search endpoint for document lists
  • 🎁 GET /api/v1/document-lists/:id Get a document list by :id

References:

Added Endpoint for a Video Import 🎁

Attention! Even when we’ve added the video endpoint already, sthe video feature is still experimental.

🎁 POST api/v1/import/videos

References:

Server Events API

We added new events to the sever events API

  • 🎁 mediaLibraryEntry.create
  • 🎁 mediaLibraryEntry.update
  • 🎁 mediaLibraryEntry.archive

References:

Internal Changes

Beta Endpoints for Publications 🎁

We added a new base endpoint /api/beta/. This allows us to expose things on the public API we are not sure enough yet, to introduce it on the v1 endpoint that we can never break.

The first 2 introduced beta endpoints are already existing and have the same format on v1, but extend the response with references. This might break in the future.

New endpoints:

  • 🎁 GET /api/beta/documents/:documentId/latestPublication
  • 🎁 GET /api/beta/documents/latestPublications

References:

Other Changes

Security

Design

Improvements

Editor

Server

Bugfixes

Patches

Livingdocs Server Patches

  • v114.0.59: chore: rerun checks
  • v114.0.58: fix(design): add new CLI command design-set-active
  • v114.0.57: fix: update integration test because of an outdated github API
  • v114.0.56: fix(hugo): require auth on all print routes
  • v114.0.55: fix(print-api): Force no-cache for print API requests

If the Livingdocs Editor is cached via CDN print API requests like getLayouts will return a cached/outdated version. This will fix the issue.

  • v114.0.54: chore(import-public-api): correctly validate publicationDate
  • v114.0.53: fix(login): Users will be able to log in even if new device emails can not be sent
  • v114.0.52: fix(print): fix crash on certificate errors
  • v114.0.51: fix(queue): Fix a typo and apply the same pending check to the xcleanup script
  • v114.0.50: fix(queue): Do not delete the consumer if we can’t transfer the pending messages
  • v114.0.49: fix: Do not send out too many ’new device login’ emails
  • v114.0.48: fix(openid-connect): fix various coercion issues
  • v114.0.47: fix(open-id-connect-sso): correctly resolve projectIds as strings
  • v114.0.46: fix(print): fix filterText() type-check

Fixes

uncaughtException: content.replace is not a function

that could occur under certain circumstances.

  • v114.0.45: fix(print): fix print host
  • v114.0.44: fix(render-pipeline): log documentId for failed renderings
  • v114.0.43: fix(list-update): finish trx if not passed in
  • v114.0.42: fix(migrations): Enable error logs for document migrations

Customers need more informations when a migration fails.

  • v114.0.41: fix(indexing): the custom indexer passes ids instead of documentIds
  • v114.0.40: fix(comments): Add maxThreadCount config property
  • v114.0.39: fix(policies): Introduce a more strict schema and allow additional properties
  • v114.0.38: fix(publish): Fix the indexing for document publish calls that are nested in a transaction
  • v114.0.37: fix(indexing): Change how we create the Redis instance in the indexing controller as it didn’t respect the sentinel config
  • v114.0.36: fix(airship): enable push notifications for web channel as well
  • v114.0.35: fix(includes): Add interaction blocker config
  • v114.0.34: fix(websocket): Rewrite the url for websockets as we do it for requests if /proxy/api is present
  • v114.0.33: fix: add data field ‘classifications’ for hugo_article
  • v114.0.32: fix(print): Handle image components
  • v114.0.31: fix(open-id-connect): correctly create users
  • v114.0.30: fix: add new npm read token
  • v114.0.29: fix: correct expiration date for cookies and accessTokens

Livingdocs Editor Patches

  • v57.33.63: chore: rerun checks

  • v57.33.62: fix(modals): allow printing articles with ctrl+p / cmd+p

  • v57.33.61: chore(ci): remove cypress from CI for dez release

  • v57.33.60: fix: update integration test because of an outdated github API

  • v57.33.59: fix(imatrics): fix styling issues leading to invisible suggestions

  • v57.33.58: fix(clipboard): render includes when dropping a component from the clipboard

  • v57.33.57: fix(viewport-viewport-units-buggyfill): improve regex to match only vh units

  • v57.33.56: fix: show an indicator incase the ES limit defaults to 10000 total documents

  • v57.33.55: fix(BW service changer): Markup updated to new standard

  • Updated BW service changer’s markup on par with recently set standard

  • v57.33.54: fix(lists): cancel spinner after error

  • v57.33.53: fix(teaser-preview): render includes in teaser preview

  • v57.33.52: fix(clipboard): fix clipboard paste for a container

  • v57.33.51: fix: alignment of component title when no description is available

  • v57.33.50: fix(comments): Show max thread count limit error

  • v57.33.49: chore(policies): Add additionalProperties: false to the policy schema to keep it in sync with the one on the server

  • v57.33.48: fix(conflict): Hide comments in conflict mode

  • v57.33.47: fix(split-pane): Minimize sidebar on conflict

  • v57.33.46: chore: Fix the livingdocs-integration.json for the release-2020-12 in bluewin

  • v57.33.45: fix(ci): Use our Docker images instead of the official Docker image

  • v57.33.44: fix(includes): Add interaction blocker config

  • v57.33.43: chore: adapt cypress tests

  • v57.33.42: fix: correctly navigate back from a custom dashboard

  • v57.33.41: fix(server): Fix support for redirecting based on x-forwarded-proto header

  • v57.33.40: fix(link tool): show linked URL when valid but not accessible via iframely link checker

  • v57.33.39: fix(media library): fix download when storage bucket is private

  • v57.33.38: fix(media library): make the context menu edit button work again

  • v57.33.37: fix: add new npm read token

  • v57.33.36: fix: make integration settings changes in the UI properly change the channelConfigDraft

Icon Legend

  • Breaking changes: 🔥
  • Feature: 🎁
  • Bugfix: 🪲
  • Chore: 🔧