### Summary
This PR is part of #564 and changes the `updateApiDocs.ts` script to
stop using a web server and a web crawler for the API docs generation.
Instead, the script uses the HTML in the artifact zip file to convert it
into markdown and copies the images from the same zip to the respective
version folder without downloading it from any web server.
### Details
#### Convert HTML to markdown
After removing the web crawler used to download the HTML files that the
script translated into markdown, we need to take into account HTML files
that are used as a redirect, like `stubs/qiskit.utils.name_args.html`
for Qiskit v0.45:
```html
<html><head><meta http-equiv="refresh" content="0; url=https://qiskit.org/documentation/apidoc/utils.html#qiskit.utils.name_args"></head></html>
```
These files were not downloaded using the web crawler and therefore, not
processed by the `sphinxHtmlToMarkdown` function. Now, the script will
try to convert those files into markdown as well, ending up in an empty
markdown file we need to remove. That is translated into this
conditional in `updateApisDocs.ts`:
```ts
if (result.markdown == "") {
continue;
}
```
#### Save images
As for the images, we don't need the web server because we can find them
in the folder called `_images` inside the artifact zip file. The script
will copy all the images to the correct API version images folder.
Moreover, the script only saves the images corresponding to the release
notes for the current APIs (not using the historical argument). This
change will allow us to remove unnecessary duplicate images we are
currently downloading.
#### Bug fix
In addition to that change, the PR fixes an underlying issue with the
old method. We were only downloading images when they were not present
in the API images folder by checking if we already had a file with the
same name. New versions of the same API have new images with the same
name as the old ones, and because of that, we were never saving the new
ones.
All the historical versions will be regenerated in a follow-up.
Commands used:
```bash
npm run gen-api -- -p qiskit -v 0.45.0 -a https://github.com/Qiskit/qiskit/actions/runs/6744953436/artifacts/1026798160
npm run gen-api -- -p qiskit-ibm-provider -v 0.7.3 -a https://github.com/Qiskit/qiskit-ibm-provider/actions/runs/7301486985/artifacts/1131430696
npm run gen-api -- -p qiskit-ibm-runtime -v 0.17.0 -a https://github.com/Qiskit/qiskit-ibm-runtime/suites/18863019852/artifacts/1100724937
```
Closes#564
---------
Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
This PR fixes the broken relative links of the Qiskit release notes. The
links are originally valid, but because we have the release notes in a
subfolder, we need to update the relative paths.
To update every link, I used the `updateApiDocs.ts` script, but added
another function to read every link and add `/api/qiskit` in front of
each relative path. This function was executed in line 220, followed by
process.exit() to avoid doing anything else.
<details>
<summary>Function to convert relative links</summary>
```typescript
async function updateLinksReleaseNotes(
projectName: string,
legacyReleaseNoteEntries: { title: string; url: string }[],
) {
for (let entry of legacyReleaseNoteEntries) {
let source = await readFile(
`${getRoot()}/docs/api/${projectName}/release-notes/${entry.title}.md`,
{ encoding: "utf8" },
);
source = transformLinks(source, (link, _) =>
link.startsWith("http") || link.startsWith("#") || link.startsWith("/")
? link
: `/api/${projectName}/${link}`,
);
await writeFile(
`${getRoot()}/docs/api/${projectName}/release-notes/${entry.title}.md`,
source,
);
}
}
```
</details>
The PR still needs to modify the `updateApiDocs.ts` to update the
relative links of the newest release notes we download when we run the
script.
Closes#386
### Summary
This PR adds a suggestion of a valid link for every broken link that the
link checker finds. This feature only works with internal links and
anchors.
To calculate the best replacement for a broken link, the function
`didYouMean` in `scripts/lib/LinkChecker.ts` uses the [Levenshtein
distance](https://en.wikipedia.org/wiki/Levenshtein_distance) between
the broken link and all the files we have in docs, as it was originally
suggested in #4.
The package used to implement the Levenshtein distance calculation is
[fast-levenshtein](https://www.npmjs.com/package/fast-levenshtein).
Relates to https://github.com/Qiskit/documentation/issues/6, which is
about docs. This PR only adds Prettier to our support code. We might
want to extend Prettier to docs in a follow up.
Having a consistent format is convenient and saves us time so that
humans don't have to manually move things around.
We have a non-trivial amount of TypeScript tooling, so this is worth
adding imo.
Bumps
[@babel/traverse](https://github.com/babel/babel/tree/HEAD/packages/babel-traverse)
from 7.23.0 to 7.23.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/babel/babel/releases"><code>@babel/traverse</code>'s
releases</a>.</em></p>
<blockquote>
<h2>v7.23.2 (2023-10-11)</h2>
<p><strong>NOTE</strong>: This release also re-publishes
<code>@babel/core</code>, even if it does not appear in the linked
release commit.</p>
<p>Thanks <a
href="https://github.com/jimmydief"><code>@jimmydief</code></a> for
your first PR!</p>
<h4>🐛 Bug Fix</h4>
<ul>
<li><code>babel-traverse</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16033">#16033</a>
Only evaluate own String/Number/Math methods (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16022">#16022</a>
Rewrite <code>.tsx</code> extension when using
<code>rewriteImportExtensions</code> (<a
href="https://github.com/jimmydief"><code>@jimmydief</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16017">#16017</a>
Fix: fallback to typeof when toString is applied to incompatible object
(<a href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>,
<code>babel-plugin-transform-modules-commonjs</code>,
<code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>,
<code>babel-runtime</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16025">#16025</a>
Avoid override mistake in namespace imports (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
<h4>Committers: 5</h4>
<ul>
<li>Babel Bot (<a
href="https://github.com/babel-bot"><code>@babel-bot</code></a>)</li>
<li>Huáng Jùnliàng (<a
href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
<li>James Diefenderfer (<a
href="https://github.com/jimmydief"><code>@jimmydief</code></a>)</li>
<li>Nicolò Ribaudo (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
<li><a
href="https://github.com/liuxingbaoyu"><code>@liuxingbaoyu</code></a></li>
</ul>
<h2>v7.23.1 (2023-09-25)</h2>
<p>Re-publishing <code>@babel/helpers</code> due to a publishing error
in 7.23.0.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/babel/babel/blob/main/CHANGELOG.md"><code>@babel/traverse</code>'s
changelog</a>.</em></p>
<blockquote>
<h2>v7.23.2 (2023-10-11)</h2>
<h4>🐛 Bug Fix</h4>
<ul>
<li><code>babel-traverse</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16033">#16033</a>
Only evaluate own String/Number/Math methods (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16022">#16022</a>
Rewrite <code>.tsx</code> extension when using
<code>rewriteImportExtensions</code> (<a
href="https://github.com/jimmydief"><code>@jimmydief</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16017">#16017</a>
Fix: fallback to typeof when toString is applied to incompatible object
(<a href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>,
<code>babel-plugin-transform-modules-commonjs</code>,
<code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>,
<code>babel-runtime</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16025">#16025</a>
Avoid override mistake in namespace imports (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b4b9942a6c"><code>b4b9942</code></a>
v7.23.2</li>
<li><a
href="b13376b346"><code>b13376b</code></a>
Only evaluate own String/Number/Math methods (<a
href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/16033">#16033</a>)</li>
<li>See full diff in <a
href="https://github.com/babel/babel/commits/v7.23.2/packages/babel-traverse">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/Qiskit/documentation/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This PR adds a link-checking script to verify internal links work. It
should be easy to extend this to check external links, and anchors
within internal links. All comments welcome.
### Details
Uses `markdown-link-extractor`, which is used by `markdown-link-check`
(one of the proposed tools).
***
First part of #4
One of the ideas from https://github.com/Qiskit/documentation/issues/63.
This is useful to me right now because I want to only redo
qiskit-ibm-provider.
`yargs` is a popular command line library in JavaScript. I've used it in
a personal project and had a great experience with it.
There will be future config we need to add, like probably having users
tell us what CI URL to look at when generating the docs, whereas right
now we pull from the live websites (see #63). I think it's likely that
config will be best set via a config file, rather than command line
arguments. But, even if we have that config file in the future, I still
think we'd want `--packages` to be a CLI arg since that is a quick thing
that you want to be flexible with setting, vs updating a file. If we
switch 100% to a file, that's fine; we can always remove `yargs`.
Part 2 of https://github.com/Qiskit/documentation/issues/9.
The final part will run this script to generate the API docs. There are
some things I want to tweak in the script like that the
qiskit-ibm-runtime site has changed URLs, but I wanted to avoid making
changes to the script in this PR so that it simply copies them over.
This PR simply copies over the API pipeline written by @axelhzf from
closed source to open source. Its only modifications are adding
copyright headers and removing unused code from utility files.
https://github.com/Qiskit/documentation/issues/63 tracks some ideas to
simplify this pipeline in follow ups, such as not needing GitHub.
Co-authored-by: Axel Hernández Ferrera <axelhzf@gmail.com>
## Background
First part of https://github.com/Qiskit/documentation/issues/9. The docs
infrastructure expects MDX files, but `qiskit`, `qiskit-ibm-provider`,
and `qiskit-ibm-runtime` use Sphinx to generate their API references
from source code. So, @axelhzf created a pipeline to convert Sphinx
output to MDX.
We are moving that pipeline to live in open source since the API
references are also open source.
## What this PR does
This adds the `lib/sphinx` code and its tests. The code is copied over
without change, other than adding copyright headers. The PR also sets up
Jest.
A follow up PR will add the actual parsing script. I split up the PRs
since this adds non-trivial infrastructure like Jest, so to have a
smaller PR to review.
This code will be deleted from the internal repository once
https://github.com/Qiskit/documentation/issues/9 is complete and the
internal repository is consuming this one.
## Why Jest?
We use Playwright in https://github.com/Qiskit/qiskit_sphinx_theme, so I
considered using that too here for consistency.
But I stuck with the internal repo's decision to use Jest because
Playwright doesn't have an equivalent to `toMatchInlineSnapshot`. I
think Playwright is awesome, but we're never going to have Playwright
integration tests in this repository since we're not testing the docs
site itself in this repo, i.e. we don't need to spin up a server. Jest
is simpler for our unit tests.
Co-authored-by: Axel Hernández Ferrera <axelhzf@gmail.com>
In the closed source repository that consumes Qiskit/docs, we have this
same check for valid file metadata. It is useful to run it here too so
that we fail more eagerly.
Like the internal code, this uses TypeScript. It makes the code more
readable and better aligned with the internal code.
---------
Co-authored-by: Axel Hernández Ferrera <axelhzf@gmail.com>
Closes https://github.com/Qiskit/docs/issues/5.
This sets up some initial infrastructure for the repository, like CI and
the folder structure from @axelhzf's proof of concept of how to open
source documentation content.
Follow up PRs will build on this infrastructure as part of
https://github.com/Qiskit/docs/milestone/1.
## Synchronizing cSpell config with closed source
This PR copies the closed source config for cSpell.
It's a bummer that this config is duplicated. I considered if we should
automate synchronizing the config file, which would need to be the
closed source repo pulling in the closed source file, since the open
source repo cannot access the closed source repo. But I think it's not
worth the complexity to automate this, at least for now.
The closed source repository will open up a PR to use a new version of
this qiskit/docs repository. That PR will run its own spellcheck. So if
there is new content with typos, the spellcheck will fail, which is a
signal that we need to update the closed source cSpell config.
If this ends up not being sustainable, then I recommend we figure out
how to automate syncing. But for now, keep it simple.
---------
Co-authored-by: Axel Hernández Ferrera <axelhzf@gmail.com>
Co-authored-by: Eric Harvey <eric@harvey.io>
Rebecca Dimock