Commit Graph

2613 Commits

Author SHA1 Message Date
Jim Ingham a37672e2db Mark the execution of stop-hooks as non-interactive.
The intention is not to allow stop-hook commands to query the
user, so this is correct.  It also works around a deadlock in
switching to the Python Session to execute python based commands
in the stop hook when the Debugger stdin is backed by a FILE *.

Differential Revision: https://reviews.llvm.org/D90332
2020-10-29 14:41:53 -07:00
Pavel Labath 8cc49bec2e [lldb] Use reverse connection method for lldb-server tests
This fixes an flakyness is all gdb-remote tests. These tests have been
(mildly) flaky since we started using "localhost" instead of 127.0.0.1
in the test suite. The reason is that lldb-server needs to create two
sockets (v4 and v6) to listen for localhost connections. The algorithm
it uses first tries to select a random port (bind(localhost:0)) for the
first address, and then bind the same port for the second one.

The creating of the second socket can fail as there's no guarantee that
port will be available -- it seems that the (linux) kernel tries to
choose an unused port for the first socket (I've had to create thousands
of sockets to reproduce this reliably), but this can apparantly fail
when the system is under load (and our test suite creates a _lot_ of
sockets).

The socket creationg operation is considered successful if it creates at
least one socket is created, but the test harness has no way of knowing
which one it is, so it can end up connecting to the wrong address.

I'm not aware of a way to atomically create two sockets bound to the
same port. One way to fix this would be to make lldb-server report the
address is it listening on instead of just the port. However, this would
be a breaking change and it's not clear to me that's worth it (the
algorithm works pretty well under normal circumstances).

Instead, this patch sidesteps that problem by using "reverse"
connections. This way, the test harness is responsible for creating the
listening socket so it can pass the address that it has managed to open.
It also results in much simpler code overall.

To preserve test coverage for the named pipe method, I've moved the
relevant code to a dedicated test. To avoid original problem, this test
passes raw addresses (as obtained by getaddrinfo(localhost)) instead of
"localhost".

Differential Revision: https://reviews.llvm.org/D90313
2020-10-29 13:49:51 +01:00
Raphael Isemann 5dc70332d5 Revert "[lldb] Explicitly use the configuration architecture when building test executables"
This reverts commit 41185226f6.

Causes TestQuoting to fail on Windows.
2020-10-22 18:42:19 +02:00
Raphael Isemann 41185226f6 [lldb] Explicitly use the configuration architecture when building test executables
The Darwin builder currently assumes in `getArchCFlags` that the passed `arch`
value is an actual string it can string.join with vendor/os/version/env strings:

```
   triple = '-'.join([arch, vendor, os, version, env])
```

However this is not true for most tests as we just pass down the `arch=None`
default value from `TestBase.build`. This causes that if we actually end up in
this function we just error out when concatenating `None` with the other actual
strings of vendor/os/version/env. What we should do instead is check that if
there is no test-specific architecture that we fall back to the configuration's
architecture value.

It seems we already worked around this in `builder.getArchSpec` by explicitly
falling back to the architecture specified in the configuration.

This patch just moves this fallback logic to the top `build` function so that it
affects all functions called from `TestBase.build`.

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D89056
2020-10-22 15:30:25 +02:00
Alexandre Ganea 89b72209ad [LLDB][TestSuite] Improve skipIfWindowsAndNonEnglish in decorators.py
Differential Revision: https://reviews.llvm.org/D89716
2020-10-19 14:28:08 -04:00
Raphael Isemann dfc7243952 [lldb] Don't add no-sandbox entitlement when running tests on simulator
It seems that if codesigning the test executables with the
`com.apple.private.security.no-sandbox` entitlement then the simulator refuses
to launch them and every test fails with `Process launch failed: process exited
with status -1 (no such process.)`.

This patch checks if we're trying to run the test suite on the simulator and
then avoids signing the executable with `no-sandbox`.

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D89052
2020-10-13 15:08:46 +02:00
Raphael Isemann 480c440f9a [lldb] Don't let dotest silently fall back to default SDK if the specified one can't be found
If the SDK name passed to dotest can't be found by `xcrun` we silently fall back
to the default SDK. This leads to rather cryptic errors being reported later on
when linking the actual test executables.

Instead just directly log and abort when this situation is encountered and
inform the user about the invalid argument.

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D89053
2020-10-13 15:07:45 +02:00
Jonas Devlieghere 360ab009e2 [lldb] Add instrumentation runtime category 2020-10-12 16:02:40 -07:00
Jonas Devlieghere e465ddac88 [lldb] Alphabetically sort test categories (NFC) 2020-10-12 16:02:40 -07:00
Alexandre Ganea 97e7fbb343 [LLDB] More Windows non-English locales fixes
This is a follow-up for https://reviews.llvm.org/D88975
2020-10-08 17:22:42 -04:00
Alexandre Ganea 79809f58b0 [LLDB] On Windows, fix tests
This patch fixes a few issues seen when running `ninja check-lldb` in a Release build with VS2017:

- Some binaries couldn't be found (such as lldb-vscode.exe), because .exe wasn't appended to the file name.
- Many tests used to fail since our installed locale is in French - the OS error messages are not emitted in English.
- Our codepage being Windows-1252, python failed to decode some error messages with accentuations.

Differential Revision: https://reviews.llvm.org/D88975
2020-10-08 11:46:59 -04:00
Jonas Devlieghere f22496a9f4 [dotest] Simplify logic to find the Python path
Simplify the logic of parsing the lldb -P output to find the python
path. This removes the special handling for the LLDB.framework case and
instead of pattern matching known errors focus on finding a directory
path that contains an __init__.py.

Differential revision: https://reviews.llvm.org/D88840
2020-10-05 19:04:33 -07:00
Dave Lee 010d7a388b [lldb/test] Catch invalid calls to expect()
Add preconditions to `TestBase.expect()` that catch semantically invalid calls
that happen to succeed anyway. This also fixes the broken callsites caught by
these checks.

This prevents the following incorrect calls:

1. `self.expect("lldb command", "some substr")`
2. `self.expect("lldb command", "assert message", "some substr")`

Differential Revision: https://reviews.llvm.org/D88792
2020-10-05 12:41:52 -07:00
Jordan Rupprecht ad865d9d10 [lldb-vscode] Allow an empty 'breakpoints' field to clear breakpoints.
Per the DAP spec for SetBreakpoints [1], the way to clear breakpoints is: `To clear all breakpoint for a source, specify an empty array.`

However, leaving the breakpoints field unset is also a well formed request (note the `breakpoints?:` in the `SetBreakpointsArguments` definition). If it's unset, we have a couple choices:

1. Crash (current behavior)
2. Clear breakpoints
3. Return an error response that the breakpoints field is missing.

I propose we do (2) instead of (1), and treat an unset breakpoints field the same as an empty breakpoints field.

[1] https://microsoft.github.io/debug-adapter-protocol/specification#Requests_SetBreakpoints

Reviewed By: wallace, labath

Differential Revision: https://reviews.llvm.org/D88513
2020-09-30 11:32:06 -07:00
Jordan Rupprecht c3193e464c [lldb/ipv6] Support running lldb tests in an ipv6-only environment.
When running in an ipv6-only environment where `AF_INET` sockets are not available, many lldb tests (mostly gdb remote tests) fail because things like `127.0.0.1` don't work there.

Use `localhost` instead of `127.0.0.1` whenever possible, or include a fallback of creating `AF_INET6` sockets when `AF_INET` fails.

Reviewed By: labath

Differential Revision: https://reviews.llvm.org/D87333
2020-09-30 11:08:41 -07:00
Jordan Rupprecht 307b7a1d66 [lldb/test] Clean up version checking.
A few fixes while trying to figure out why tests are being skipped for arsenm:

- We check `$compiler -v`, but `-v` is `--verbose`, not `--version`. Use the long flag name.
- We check all lines matching `version ...`, but we should exit early for the first version string we see (which should be the main one). I'm not sure if this is the issue, but perhaps this is causing some users to skip some tests if another "version ..." is showing up later.
- Having `\.` in a python string is triggering pylint warnings, because it should be escaped as a regex string, e.g. `r'\.' However, `.` in a character class does not need to be escaped, as it matches only a literal `.` in that context.

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D88051
2020-09-21 16:19:28 -07:00
Jordan Rupprecht dd67581407 [lldb/test] Enable faulthandler in dotest
Register the `faulthandler` module so we can see what lldb tests are doing when they misbehave (e.g. run under a test runner that sets a timeout). This will print a stack trace for the following signals:

- `SIGSEGV`, `SIGFPE`, `SIGABRT`, `SIGBUS`, and `SIGILL` (via `faulthandler.enable()`)
- `SIGTERM` (via `faulthandler.register(SIGTERM)`) [This is what our test runners sends when it times out].

The only signal we currently handle is `SIGINT` (via `unittest2.signals.installHandler()`) so there should be no overlap added by this patch.

Because this import is not available until python3, and the `register()` method is not available on Windows, this is enabled defensively.

This should have absolutely no effect when tests are passing (or even normally failing), but can be observed by running this while ninja is running:

```
kill -s SIGTERM $(ps aux | grep dotest.py | head -1 | awk '{print $2}')
```

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D87637
2020-09-16 14:26:40 -07:00
Walter Erquinigo 132e57bc59 Retry of D84974
- Fix a small issue caused by a conflicting name (GetObject) on Windows.
  The fix was to rename the internal GetObject function to
  GetNextFunction.
2020-09-14 10:44:13 -07:00
Jonas Devlieghere 2955a27abc [lldb] Pass the arch as part of the triple in the ARCH_CFLAGS 2020-09-09 14:41:25 -07:00
Walter Erquinigo 5c463d107d Revert "Retry of D84974"
This reverts commit 5b2b4f331d.

This caused a link error in
http://lab.llvm.org:8011/builders/lldb-x64-windows-ninja/builds/18794/steps/build/logs/stdio
2020-09-08 13:41:11 -07:00
Walter Erquinigo 5b2b4f331d Retry of D84974
The test is being disabled on Linux, as lldb-vscode has a bug with
--wait-for on LInux.
I'm also fixing some compilation warnings.
2020-09-08 11:50:09 -07:00
Muhammad Omair Javaid 9bee13f890 Move targetHasSVE function to lldbtest.py
targetHasSVE helper function was added to test for availability of SVE support
by connected platform. We now intend to use this function in other testcases
and I am moving it to a generic location in lldbtest.py to allow usage by
other upcoming testcases.

Reviewed By: labath

Differential Revision: https://reviews.llvm.org/D86872
2020-09-07 08:37:39 +05:00
David Spickett 9f18f3c858 [lldb] Improve test failure reporting for expect()
This updates the errors reported by expect()
to something like:

```
Ran command:
"help"

Got output:
Debugger commands:
<...>

Expecting start string: "Debugger commands:" (was found)
Expecting end string: "foo" (was not found)
```
(see added tests for more examples)

This shows the user exactly what was run,
what checks passed and which failed. Along with
whether that check was supposed to pass.
(including what regex patterns matched)

These lines are also output to the test
trace file, whether the test passes or not.

Note that expect() will still fail at the first failed
check, in line with previous behaviour.

Also I have flipped the wording of the assert
message functions (.*_MSG) to describe failures
not successes. This makes more sense as they are
only shown on assert failures.

Reviewed By: labath

Differential Revision: https://reviews.llvm.org/D86792
2020-09-03 13:35:05 +01:00
Walter Erquinigo 5f6ca065a5 Revert de6caf871b and 51128b670d (https://reviews.llvm.org/D84974)
The tests seem to be timing out in all linux bots. Need further analysis.

Revert "run in terminal"

This reverts commit de6caf871b.
2020-09-02 17:06:48 -07:00
Walter Erquinigo de6caf871b run in terminal 2020-09-02 14:38:00 -07:00
Raphael Isemann b51321ccc8 [lldb] Fix TestCompletion's pid completion failing randomly
TestCompletion is randomly failing on some bots. The error message however states
that the computed completions actually do contain the expected pid we're
looking for, so there shouldn't be any test failure.

The reason for that turns out to be that complete_from_to is actually used
for testing two different features. It can be used for testing what the
common prefix for the list of completions is and *also* for checking all the
possible completions that are returned for a command. Which one of the two
things should be checked can't be defined by a parameter to the function, but
is instead guessed by the test method instead based on the results that were
returned. If there is a common prefix in all completions, then that prefix
is searched and otherwise all completions are searched.

For TestCompletion's pid test this behaviour leads to the strange test failures.
If all the pid's that our test LLDB can see have a common prefix (e.g., it
can only see pids [123, 122, 10004, 10000] -> common prefix '1'), then
complete_from_to check that the common prefix contains our pid, which is
always fails ('1' doesn't contain '123' or any other valid pid). If there
isn't a common prefix (e.g., pids are [123, 122, 10004, 777]) then
complete_from_to will check the list of completions instead which works correctly.

This patch is fixing this by adding a simple check method that doesn't
have this behaviour and is simply searching the returned list of completions.
This should get the bots green while I'm working on a proper fix that fixes
complete_from_to.
2020-08-31 12:22:41 +02:00
Jonas Devlieghere 141c8475b6 [lldb] Get rid of LLDB_LIB_DIR and LLDB_IMPLIB_DIR in dotest
This patch removes the rather confusing LLDB_LIB_DIR and LLDB_IMPLIB_DIR
environment variables. They are confusing because LLDB_LIB_DIR would
point to the bin subdirectory in the build root while LLDB_IMPLIB_DIR
would point to the lib subdirectory. The reason far this was
LLDB.framework, which gets build under bin.

This patch replaces their uses with configuration.lldb_framework_path
and configuration.lldb_libs_dir respectively.

Differential revision: https://reviews.llvm.org/D86817
2020-08-28 15:45:54 -07:00
Jordan Rupprecht 031554ed46 Reland "[test] Exit with an error if no tests are run."
This reverts commit a06c28df3e (reland adb5c23f8c).

The issue with PExpect tests on Windows should be fixed with e5e05ecf65.
2020-08-28 14:27:37 -07:00
Jordan Rupprecht e5e05ecf65 [lldb/test] Use @skipIfWindows for PExpectTest
Annotating `PExpectTest` with `@skipIfWindows` instead of marking it as an empty class will make the test runner recognize it as a test class, which should allow me to reland adb5c23f8c.

I don't have a windows machine to verify this works, but I did some tests using `@skipIfLinux` and they all worked as expected. In case the `pexpect` import is not at all available on windows, I moved it to within the method where it's used.

Reviewed By: labath

Differential Revision: https://reviews.llvm.org/D86745
2020-08-28 11:41:07 -07:00
Jonas Devlieghere b981924bdd [lldb] Move triple construction out of getArchCFlags in DarwinBuilder (NFC)
Move the construction of the triple out of getArchCFlags in the
DarwinBuilder.
2020-08-27 09:31:01 -07:00
David Spickett 9ad5d37fd9 [lldb] Correct wording of EXP_MSG
EXP_MSG generates a message to show on assert
failure. Currently it looks like:
AssertionError: False is not True : '<cmd>'
returns expected result, got '<actual output>'

Which seems to say that the test failed but
also got the expected result.

It should say:
AssertionError: False is not True : '<cmd>'
returned unexpected result, got '<actual output>'

Reviewed By: teemperor, #lldb

Differential Revision: https://reviews.llvm.org/D86603
2020-08-26 11:52:30 +01:00
Jonas Devlieghere d3a49b03a5 [lldb] Remove --rerun-all-issues as its functionality no longer exists
The logic behind --rerun-all-issues was removed when we switched to LIT
as the test driver. This patch just removes the dotest option and
corresponding entry in configuration.py.
2020-08-21 14:28:08 -07:00
Jonas Devlieghere a6eb70c052 [lldb] Return empty string from getExtraMakeArgs when not implemented
No return statement means the method returns None which breaks a list
comprehension down the line that expects a str instance.
2020-08-19 17:52:50 -07:00
Jonas Devlieghere a3fc61c80f [lldb] Move Xcode SDK helper functions into lldbutil
This allows the logic to be reused by both the builders and the tests.
2020-08-19 13:30:27 -07:00
Jonas Devlieghere b40a3814b6 [lldb] Code sign binaries with entitlements
Binaries need to be code signed with entitlements to run on device.

Differential revision: https://reviews.llvm.org/D86237
2020-08-19 11:55:36 -07:00
Jonas Devlieghere e5d08fcbac [lldb] Extend Darwin builder to pass the ARCH_CFLAGS spec to Make.
Construct the ARCH_CFLAGS in Python rather than in Make by disassembling
the TRIPLE.

Differential revision: https://reviews.llvm.org/D85539
2020-08-19 11:47:29 -07:00
Jonas Devlieghere 074c591a7e [lldb] Add getExtraMakeArgs to Builder (NFC)
Instead of a new method for each variable any subclass might want to
set, have a method getExtraMakeArgs that each subclass can use to return
whatever extra Make arguments it wants.

As per Pavel's suggestion in D85539.
2020-08-19 09:47:25 -07:00
Jonas Devlieghere 804691adc9 [lldb] Fix buildDsym signature in Builder base class
The method was missing the optional argument `testname`.
2020-08-19 09:47:25 -07:00
Jonas Devlieghere b623f3c0b4 [lldb] Move builders under lldbsuite.test as they import lldbtest (NFC) 2020-08-19 09:07:51 -07:00
Jonas Devlieghere 1922bf12e1 [lldb] Convert builders to use inheritance (NFC)
Rather than have different modules for different platforms, use
inheritance so we can have a Builer base class and optional child
classes that override platform specific methods.

Differential revision: https://reviews.llvm.org/D86174
2020-08-19 08:44:29 -07:00
Pavel Labath d7363397c6 [lldb] Add typedefs to the DeclContext they are created in
TypeSystemClang::CreateTypedef was creating a typedef in the right
DeclContext, but it was not actually adding it as a child of the
context. The resulting inconsistent state meant that we would be unable
to reference the typedef from an expression directly, but we could use
them if they end up being pulled in by some previous subexpression
(because the ASTImporter will set up the correct links in the expression
ast).

This patch adds the typedef to the decl context it is created in.

Differential Revision: https://reviews.llvm.org/D86140
2020-08-19 14:57:43 +02:00
Jonas Devlieghere 514bcb325d [lldb] Remove unused function getArchFlag (NFC) 2020-08-18 15:20:57 -07:00
Jonas Devlieghere e9b0994012 [lldb] Replace unittest2.expectedFailure with expectedFailure (NFC)
Rename the existing expectedFailure to expectedFailureIfFn to better
describe its purpose and provide an overload for
unittest2.expectedFailure in decorators.py.
2020-08-17 10:05:49 -07:00
Raphael Isemann c2f9454a16 [lldb] Add SBModule::GarbageCollectAllocatedModules and clear modules after each test run
Right now the only places in the SB API where lldb:: ModuleSP instances are
destroyed are in SBDebugger::MemoryPressureDetected (where it's just attempted
but not guaranteed) and in SBDebugger::DeleteTarget (which will be removed in
D83933). Tests that directly create an lldb::ModuleSP and never create a target
therefore currently leak lldb::Module instances. This triggers the sanity checks
in lldbtest that make sure that the global module list is empty after a test.

This patch adds SBModule::GarbageCollectAllocatedModules as an explicit way to
clean orphaned lldb::ModuleSP instances. Also we now start calling this method
at the end of each test run and move the sanity check behind that call to make
this work. This way even tests that don't create targets can pass the sanity
check.

This fixes TestUnicodeSymbols.py when D83865 is applied (which makes that the
sanity checks actually fail the test).

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D83876
2020-08-17 11:00:19 +02:00
Raphael Isemann 24fc3177c1 [lldb] Print the exception traceback when hitting cleanup errors
Right now if the test suite encounters a cleanup error it just prints "CLEANUP
ERROR:" but not any additional information.

This patch just prints the exception that caused the cleanup error. This should
make debugging the failing tests for D83865 easier (and seems in general nice to
have).

Reviewed By: labath

Differential Revision: https://reviews.llvm.org/D83874
2020-08-17 09:53:52 +02:00
Raphael Isemann 46ed27ff1b [lldb] Make packetlog_get_dylib_info returns the last full response
In sanitized builds the last packet this function finds for the
TestMacCatalyst and TestPlatformSimulator tests is for the asan runtime.

```
 <  69> send packet: $jGetLoadedDynamicLibrariesInfos:{"solib_addresses":[4296048640]}]#3a <
 715> read packet: ${"images":[{"load_address":4296048640,"mod_date":0,"pathname":
 "/Users/buildslave/jenkins/workspace/lldb-cmake-sanitized/host-compiler/lib/clang/12.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib",
 "uuid":"8E38A2CD-753F-3E0F-8EB0-F4BD5788A5CA",
 "min_version_os_name":"macosx","min_version_os_sdk":"10.9",
 "mach_header":{"magic":4277009103,"cputype":16777223,"cpusubtype":3,"filetype":6,
 "flags":43090053}],"segments":[{"name":"__TEXT","vmaddr":0,"vmsize":565248,"fileoff":0,
 "filesize":565248,"maxprot":5}],{"name":"__DATA","vmaddr":565248,"vmsize":13152256,"fileoff":565248,
 "filesize":20480,"maxprot":3}],{"name":"__LINKEDIT","vmaddr":13717504,"vmsize":438272,"fileoff":585728,
 "filesize":435008,"maxprot":1}]]}]]}]#00
```

This just fetches the last package which has fetch_all_solibs and we know
it will contain the image of our test executable to get the tests running again.
2020-08-14 14:51:13 +02:00
Pavel Labath e6b1b61054 [lldb] Fix py3 incompatibility in gdbremote_testcase.py
This didn't cause test failures since this variable is only used during
connection shutdown.
2020-08-14 12:15:25 +02:00
Shu Anzai de9e85026f [lldb] Display autosuggestion part in gray if there is one possible suggestion
This is relanding D81001. The patch originally failed as on newer editline
versions it seems CC_REFRESH will move the cursor to the start of the line via
\r and then back to the original position. On older editline versions like
the one used by default on macOS, CC_REFRESH doesn't move the cursor at all.
As the patch changed the way we handle tab completion (previously we did
REDISPLAY but now we're doing CC_REFRESH), this caused a few completion tests
to receive this unexpected cursor movement in the output stream.
This patch updates those tests to also accept output that contains the specific
cursor movement commands (\r and then \x1b[XC). lldbpexpect.py received an
utility method for generating the cursor movement escape sequence.

Original summary:

I implemented autosuggestion if there is one possible suggestion.
I set the keybinds for every character. When a character is typed, Editline::TypedCharacter is called.
Then, autosuggestion part is displayed in gray, and you can actually input by typing C-k.
Editline::Autosuggest is a function for finding completion, and it is like Editline::TabCommand now, but I will add more features to it.

Testing does not work well in my environment, so I can't confirm that it goes well, sorry. I am dealing with it now.

Reviewed By: teemperor, JDevlieghere, #lldb

Differential Revision: https://reviews.llvm.org/D81001
2020-08-14 11:37:49 +02:00
Jonas Devlieghere 180d6ed667 [lldb] Skip TestStepScripted with reproducers
Some of the test methods were already skipped because of an unexpected
packet. The test started failing after it was expanded. Skip the whole
test with reproducers so we don't have to add the decorator for every
method.
2020-08-13 09:46:52 -07:00
Raphael Isemann dd0fdf8030 [lldb] Add support for checking children in expect_expr
expect_expr currently can't verify the children of the result SBValue.

This patch adds the ability to check them. The idea is to have a CheckValue
class where one can specify what attributes of a SBValue should be checked.
Beside the properties we already check for (summary, type, etc.) this also
has a list of children which is again just a list of CheckValue object (which
can also have children of their own).

The main motivation is to make checking the children no longer based
on error-prone substring checks that allow tests to pass just because
for example the error message contains the expected substrings by accident.

I also expect that we can just have a variant of `expect_expr` for LLDB's
expression paths (aka 'frame var') feature.

Reviewed By: labath

Differential Revision: https://reviews.llvm.org/D83792
2020-08-12 12:11:24 +02:00