The idea behind this PR is to generate logs in a structured-json
format, such that we can use automated tooling to parse and debug
O2 logs as well.
If logs are to be written and analysed in files, one can enable:
`ZO_LOG_FILE_DIR=/tmp/openobserve`
The output file will be of format -
`/tmp/openobserve/o2.log.2023-12-18-10`
and it will be rotated daily.
And add a new environment `ZO_LOG_JSON_FORMAT=false`, set to `true` to
enable json format.
---------
Co-authored-by: Ankur Srivastava <ankur@grover.com>
Co-authored-by: hengfeiyang <hengfei.yang@gmail.com>
This PR implements the first draft for the Real User Monitoring.
The endpoints at the moment are:
- `/rum/v1/{org_id}/rum`
- `/rum/v1/{org_id}/replay`
- `/rum/v1/{org_id}/logs`
For now the authentication is done using `oo-api-key` which is
being passed in the query-param using `browser-sdk`.
As of now, the data is being collected in the following three streams:
- `_rumdata` for rum data
- `_rumlog` for log data
- `_sessionreplay` for session-replay data
The data is being ingested using the multi-json implementation, with a
small change where some extra data is also ingested in the json from
the headers, query-params etc. using a middleware extractor and stores
the data in `HashMap<String, String>` and gets ingested in this
multi-json.
A file containing regexes for common user-agents is also committed which
is read during the initialization time which is used to parse incoming
user-agents and browser, os etc are inserted into the incoming row of
data.
Added the endpoints for `get`, `put`, `post` for rum-tokens.
- `/api/{org_id}/organizations/rumtoken`
#### maxmind integration
Integrated the maxmind-db data in the source. Currently the data
is injected via the middleware and the file gets uploaded every
24 hours.
The file pointer gets mutated and updated when there is a change
in the mmdb file. To efficiently do this, we also query the sha256
of our file and compare it using the .sha256 file available on the
public bucket.
## Implement
- [x] Implement file_list API for gRPC
- [x] Debuging deadlock issue
## Todo
- [ ] Querier should use the file_list API for search
---------
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Auto upgrade from v0.5.2 to v0.6.0. Moves from sled to sqlite for local mode without user intervention. There should be no noticeable changes to end users.
Openobserve uses vrl & vector from vector fork. Vrl is available as
crate , we can use the crate as dependancy rather than using it from our
fork .
This also fixes https://github.com/openobserve/openobserve/issues/1233
Subhra264