This patch introduces a new type of ScriptedProcess: CrashLogScriptedProcess.
It takes advantage of lldb's crashlog parsers and Scripted Processes to
reconstruct a static debugging session with symbolicated stackframes, instead
of just dumping out everything in the user's terminal.
The crashlog command also has an interactive mode that only provide a
very limited experience. This is why this patch removes all the logic
for this interactive mode and creates CrashLogScriptedProcess instead.
This will fetch and load all the libraries that were used by the crashed
thread and re-create all the frames artificially.
rdar://88721117
Differential Revision: https://reviews.llvm.org/D119501
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
Update `__init__.py` generation to implement `__lldb_init_module`, which calls
`__lldb_init_module` on submodules that define it.
This allows the use case where a user runs `command script import lldb.macosx`.
With this change, the `__lldb_init_module` function in `crashlog.py` and
`heap.py` will be run, which is where command registration is occurring.
Differential Revision: https://reviews.llvm.org/D119179
Return our PythonObject wrappers instead of raw PyObjects (obfuscated as
void *). This ensures that ownership (reference counts) of python
objects is automatically tracked.
Differential Revision: https://reviews.llvm.org/D117462
The lldbconfig module was necessary to run the LLDB test suite against a
reproducer. Since this functionality has been removed, the module is no
longer necessary.
Unlike the rest of our SB objects, SBEvent and SBCommandReturnObject
have the ability to hold non-owning pointers to their non-SB
counterparts. This makes it hard to ensure the SB objects do not become
dangling once their backing object goes away.
While we could make these two objects behave like others, that would
require plubming even more shared pointers through our internal code
(Event objects are mostly prepared for it, CommandReturnObject are not).
Doing so seems unnecessarily disruptive, given that (unlike for some of
the other objects) I don't see any good reason why would someone want to
hold onto these objects after the function terminates.
For that reason, this patch implements a different approach -- the SB
objects will still hold non-owning pointers, but they will be reset to
the empty/default state as soon as the function terminates. This python
code will not crash if the user decides to store these objects -- but
the objects themselves will be useless/empty.
Differential Revision: https://reviews.llvm.org/D116162
This starts to fix the other half of the lifetime problems in this code
-- dangling references. SB objects created on the stack will go away
when the function returns, which is a problem if the python code they
were meant for stashes a reference to them somewhere. Most of the time
this goes by unnoticed, as the code rarely has a reason to store these,
but in case it does, we shouldn't respond by crashing.
This patch fixes the management for a couple of SB objects (Debugger,
Frame, Thread). The SB objects are now created on the heap, and
their ownership is immediately passed on to SWIG, which will ensure they
are destroyed when the last python reference goes away. I will handle
the other objects in separate patches.
I include one test which demonstrates the lifetime issue for SBDebugger.
Strictly speaking, one should create a test case for each of these
objects and each of the contexts they are being used. That would require
figuring out how to persist (and later access) each of these objects.
Some of those may involve a lot of hoop-jumping (we can run python code
from within a frame-format string). I don't think that is
necessary/worth it since the new wrapper functions make it very hard to
get this wrong.
Differential Revision: https://reviews.llvm.org/D115925
I've found my recent ventures into the swig land painful because
of the strange way they are formatted. This patch attempts to alleviate
future headaches by formatting these files into something resembling the
normal llvm style.
Unfortunately, completely formatting these files automatically does not
work because clang format gets confused by swigs % syntax, so I have
employed a hybrid approach where I formatted blocks of c++ code with
clang-format and then manually massaged the code until it looked
reasonable (and compiled).
I don't expect these files to remain perfectly formatted (although, if
one's editor is configured to configure the current line/block on
request, one can get pretty good results by using it judiciously), but
at least it will prevent the (mangled form of the) old lldb style being
proliferated endlessly.
Differential Revision: https://reviews.llvm.org/D115736
StructuredDataImpl ownership semantics is unclear at best. Various
structures were holding a non-owning pointer to it, with a comment that
the object is owned somewhere else. From what I was able to gather that
"somewhere else" was the SBStructuredData object, but I am not sure that
all created object eventually made its way there. (It wouldn't matter
even if they did, as we are leaking most of our SBStructuredData
objects.)
Since StructuredDataImpl is just a collection of two (shared) pointers,
there's really no point in elaborate lifetime management, so this patch
replaces all StructuredDataImpl pointers with actual objects or
unique_ptrs to it. This makes it much easier to resolve SBStructuredData
leaks in a follow-up patch.
Differential Revision: https://reviews.llvm.org/D114791
Some pythons are configured to set platlib somewhere outside of their
sys.prefix. It's important that we at least use some reasonable
default for LLDB_PYTHON_RELATIVE_PATH even in that case, because
even if the user overrides it on the cmake invocation, cmake will
still be called without the override in order to build tablegen.
Reviewed By: JDevlieghere, clayborg
Differential Revision: https://reviews.llvm.org/D114973
The LLDBSWIGPython functions had (at least) two problems:
- There wasn't a single source of truth (a header file) for the
prototypes of these functions. This meant that subtle differences
in copies of function declarations could go by undetected. And
not-so-subtle differences would result in strange runtime failures.
- All of the declarations had to have an extern "C" interface, because
the function definitions were being placed inside and extert "C" block
generated by swig.
This patch fixes both problems by moving the function definitions to the
%header block of the swig files. This block is not surrounded by extern
"C", and seems more appropriate anyway, as swig docs say it is meant for
"user-defined support code" (whereas the previous %wrapper code was for
automatically-generated wrappers).
It also puts the declarations into the SWIGPythonBridge header file
(which seems to have been created for this purpose), and ensures it is
included by all code wishing to define or use these functions. This
means that any differences in the declaration become a compiler error
instead of a runtime failure.
Differential Revision: https://reviews.llvm.org/D114369
Configuring lldb with `LLDB_ENABLE_PYTHON=OFF` and `LLDB_ENABLE_LUA=ON` results in a CMake error:
CMake Error at lldb/bindings/lua/CMakeLists.txt:47 (create_relative_symlink):
Unknown CMake command "create_relative_symlink".
Call Stack (most recent call first):
lldb/CMakeLists.txt:117 (finish_swig_lua)
This is because the CMake function `create_relative_symlink` only exists in `lldb/bindings/python/CMakeLists.txt`, and not in `lldb/bindings/lua/CMakeLists.txt`.
Move the function to `lldb/bindings/CMakeLists.txt`, so it is available for all language bindings.
Reviewed By: labath
Differential Revision: https://reviews.llvm.org/D114465
Using an lldb_private object in the bindings involves three steps
- wrapping the object in it's lldb::SB variant
- using swig to convert/wrap that to a PyObject
- wrapping *that* in a lldb_private::python::PythonObject
Our SBTypeToSWIGWrapper was only handling the middle part. This doesn't
just result in increased boilerplate in the callers, but is also a
functionality problem, as it's very hard to get the lifetime of of all
of these objects right. Most of the callers are creating the SB object
(step 1) on the stack, which means that we end up with dangling python
objects after the function terminates. Most of the time this isn't a
problem, because the python code does not need to persist the objects.
However, there are legitimate cases where they can do it (and even if
the use case is not completely legitimate, crashing is not the best
response to that).
For this reason, some of our code creates the SB object on the heap, but
it has another problem -- it never gets cleaned up.
This patch begins to add a new function (ToSWIGWrapper), which does all
of the three steps, while properly taking care of ownership. In the
first step, I have converted most of the leaky code (except for
SBStructuredData, which needs a bit more work).
Differential Revision: https://reviews.llvm.org/D114259
This file was way more complicated than it needed to be.
This patch removes the automagic reference-to-pointer delegation and
replaces the template specializations with regular free functions
(taking reference arguments).
The reason I chose references is twofold:
- there are more arguments being passed by reference than by pointer
- the reference arguments make it more obvious that there is a lot of
leaking going on in there.
Currently, the code was assuming that the pointer arguments have some
kind of a special meaning and that pointer functions take ownership of
their arguments, which isn't true (it's possible it was true at some
point in the past, I haven't done the archeology).
This makes it easier to implement proper lifetime management in
follow-up patches.
Differential Revision: https://reviews.llvm.org/D114150
distutils is deprecated and will be removed, so we shouldn't be
using it.
We were using it to compute LLDB_PYTHON_RELATIVE_PATH.
Discussing a similar issue
[at python.org](https://bugs.python.org/issue41282), Filipe Laíns said:
If you are relying on the value of distutils.sysconfig.get_python_lib()
as you shown in your system, you probably don't want to. That
directory (dist-packages) should be for Debian provided packages
only, so moving to sysconfig.get_path() would be a good thing,
as it has the correct value for user installed packages on your
system.
So I propose using a relative path from `sys.prefix` to
`sysconfig.get_path("platlib")` instead.
On Mac and windows, this results in the same paths as we had before,
which are `lib/python3.9/site-packages` and `Lib\site-packages`,
respectively.
On ubuntu however, this will change the path from
`lib/python3/dist-packages` to `lib/python3.9/site-packages`.
This change seems to be correct, as Filipe said above, `dist-packages`
belongs to the distribution, not us.
Reviewed By: labath
Differential Revision: https://reviews.llvm.org/D114106
see: https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/38387/console
```
Could not find a relative path to sys.executable under sys.prefix
tried: /usr/local/opt/python/bin/python3.7
tried: /usr/local/opt/python/bin/../Frameworks/Python.framework/Versions/3.7/bin/python3.7
sys.prefix: /usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7
```
It was unable to find LLDB_PYTHON_EXE_RELATIVE_PATH because it was not resolving
the real path of sys.prefix.
caused by: https://reviews.llvm.org/D113650
LLDB doesn't use only the python stable ABI, which means loading
it into an incompatible python can cause the process to crash.
_lldb.so should be named with the full EXT_SUFFIX from sysconfig
-- such as _lldb.cpython-39-darwin.so -- so this doesn't happen.
Reviewed By: JDevlieghere
Differential Revision: https://reviews.llvm.org/D112972
Apparently "{sys.prefix}/bin/python3" isn't where you find the
python interpreter on windows, so the test I wrote for
-print-script-interpreter-info is failing.
We can't rely on sys.executable at runtime, because that will point
to lldb.exe not python.exe.
We can't just record sys.executable from build time, because python
could have been moved to a different location.
But it should be OK to apply relative path from sys.prefix to sys.executable
from build-time to the sys.prefix at runtime.
Reviewed By: JDevlieghere
Differential Revision: https://reviews.llvm.org/D113650
It is surprisingly difficult to write a simple python script that
can reliably `import lldb` without failing, or crashing. I'm
currently resorting to convolutions like this:
def find_lldb(may_reexec=False):
if prefix := os.environ.get('LLDB_PYTHON_PREFIX'):
if os.path.realpath(prefix) != os.path.realpath(sys.prefix):
raise Exception("cannot import lldb.\n"
f" sys.prefix should be: {prefix}\n"
f" but it is: {sys.prefix}")
else:
line1, line2 = subprocess.run(
['lldb', '-x', '-b', '-o', 'script print(sys.prefix)'],
encoding='utf8', stdout=subprocess.PIPE,
check=True).stdout.strip().splitlines()
assert line1.strip() == '(lldb) script print(sys.prefix)'
prefix = line2.strip()
os.environ['LLDB_PYTHON_PREFIX'] = prefix
if sys.prefix != prefix:
if not may_reexec:
raise Exception(
"cannot import lldb.\n" +
f" This python, at {sys.prefix}\n"
f" does not math LLDB's python at {prefix}")
os.environ['LLDB_PYTHON_PREFIX'] = prefix
python_exe = os.path.join(prefix, 'bin', 'python3')
os.execl(python_exe, python_exe, *sys.argv)
lldb_path = subprocess.run(['lldb', '-P'],
check=True, stdout=subprocess.PIPE,
encoding='utf8').stdout.strip()
sys.path = [lldb_path] + sys.path
This patch aims to replace all that with:
#!/usr/bin/env lldb-python
import lldb
...
... by adding the following features:
* new command line option: --print-script-interpreter-info. This
prints language-specific information about the script interpreter
in JSON format.
* new tool (unix only): lldb-python which finds python and exec's it.
Reviewed By: JDevlieghere
Differential Revision: https://reviews.llvm.org/D112973
This patch changes the `ScriptedThread` initializer in couple of ways:
- It replaces the `SBTarget` parameter by a `SBProcess` (pointing to the
`ScriptedProcess` that "owns" the `ScriptedThread`).
- It adds a reference to the `ScriptedProcessInfo` Dictionary, to pass
arbitrary user-input to the `ScriptedThread`.
This patch also fixes the SWIG bindings methods that call the
`ScriptedProcess` and `ScriptedThread` initializers by passing all the
arguments to the appropriate `PythonCallable` object.
Differential Revision: https://reviews.llvm.org/D112046
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
This patch adds support for memory regions in Scripted Processes.
This is necessary to read the stack memory region in order to
reconstruct each stackframe of the program.
In order to do so, this patch makes some changes to the SBAPI, namely:
- Add a new constructor for `SBMemoryRegionInfo` that takes arguments
such as the memory region name, address range, permissions ...
This is used when reading memory at some address to compute the offset
in the binary blob provided by the user.
- Add a `GetMemoryRegionContainingAddress` method to `SBMemoryRegionInfoList`
to simplify the access to a specific memory region.
With these changes, lldb is now able to unwind the stack and reconstruct
each frame. On top of that, reloading the target module at offset 0 allows
lldb to symbolicate the `ScriptedProcess` using debug info, similarly to an
ordinary Process.
To test this, I wrote a simple program with multiple function calls, ran it in
lldb, stopped at a leaf function and read the registers values and copied
the stack memory into a binary file. These are then used in the python script.
Differential Revision: https://reviews.llvm.org/D108953
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
This patch introduces the `ScriptedThread` class with its python
interface.
When used with `ScriptedProcess`, `ScriptedThreaad` can provide various
information such as the thread state, stop reason or even its register
context.
This can be used to reconstruct the program stack frames using lldb's unwinder.
rdar://74503836
Differential Revision: https://reviews.llvm.org/D107585
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
This patch splits the previous `ScriptedProcessPythonInterface` into
multiple specific classes:
1. The `ScriptedInterface` abstract class that carries the interface
instance object and its virtual pure abstract creation method.
2. The `ScriptedPythonInterface` that holds a generic `Dispatch` method that
can be used by various interfaces to call python methods and also keeps a
reference to the Python Script Interpreter instance.
3. The `ScriptedProcessInterface` that describes the base Scripted
Process model with all the methods used in the underlying script.
All these components are used to refactor the `ScriptedProcessPythonInterface`
class, making it more modular.
This patch is also a requirement for the upcoming work on `ScriptedThread`.
Differential Revision: https://reviews.llvm.org/D107521
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
This patch introduces Scripted Processes to lldb.
The goal, here, is to be able to attach in the debugger to fake processes
that are backed by script files (in Python, Lua, Swift, etc ...) and
inspect them statically.
Scripted Processes can be used in cooperative multithreading environments
like the XNU Kernel or other real-time operating systems, but it can
also help us improve the debugger testing infrastructure by writting
synthetic tests that simulates hard-to-reproduce process/thread states.
Although ScriptedProcess is not feature-complete at the moment, it has
basic execution capabilities and will improve in the following patches.
rdar://65508855
Differential Revision: https://reviews.llvm.org/D95713
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
This patch introduces Scripted Processes to lldb.
The goal, here, is to be able to attach in the debugger to fake processes
that are backed by script files (in Python, Lua, Swift, etc ...) and
inspect them statically.
Scripted Processes can be used in cooperative multithreading environments
like the XNU Kernel or other real-time operating systems, but it can
also help us improve the debugger testing infrastructure by writting
synthetic tests that simulates hard-to-reproduce process/thread states.
Although ScriptedProcess is not feature-complete at the moment, it has
basic execution capabilities and will improve in the following patches.
rdar://65508855
Differential Revision: https://reviews.llvm.org/D95713
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
In order to facilitate the writting of Scripted Processes, this patch
introduces a `ScriptedProcess` python base class in the lldb module.
The base class holds the python interface with all the - abstract -
methods that need to be implemented by the inherited class but also some
methods that can be overwritten.
This patch also provides an example of a Scripted Process with the
`MyScriptedProcess` class.
rdar://65508855
Differential Revision: https://reviews.llvm.org/D95712
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
This patch adds a ScriptedProcess interface to the ScriptInterpreter and
more specifically, to the ScriptInterpreterPython.
This interface will be used in the C++ `ScriptProcess` Process Plugin to
call the script methods.
At the moment, not all methods are implemented, they will upstreamed in
upcoming patches.
This patch also adds helper methods to the ScriptInterpreter to
convert `SBAPI` Types (SBData & SBError) to `lldb_private` types
(DataExtractor & Status).
rdar://65508855
Differential Revision: https://reviews.llvm.org/D95711
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
Ignore `-Wreturn-type-c-linkage` diagnostics for `LLDBSwigPythonBreakpointCallbackFunction`.
The function is defined in `python-wrapper.swig` which uses `extern "C" { ... }` blocks.
The declaration of this function in `ScriptInterpreterPython.cpp` already uses these
same pragmas to silence the warning there.
This prevents `-Werror` builds from failing.
Differential Revision: https://reviews.llvm.org/D98368
This patch introduces Scripted Processes to lldb.
The goal, here, is to be able to attach in the debugger to fake processes
that are backed by script files (in Python, Lua, Swift, etc ...) and
inspect them statically.
Scripted Processes can be used in cooperative multithreading environments
like the XNU Kernel or other real-time operating systems, but it can
also help us improve the debugger testing infrastructure by writting
synthetic tests that simulates hard-to-reproduce process/thread states.
Although ScriptedProcess is not feature-complete at the moment, it has
basic execution capabilities and will improve in the following patches.
rdar://65508855
Differential Revision: https://reviews.llvm.org/D95713
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
In order to facilitate the writting of Scripted Processes, this patch
introduces a `ScriptedProcess` python base class in the lldb module.
The base class holds the python interface with all the - abstract -
methods that need to be implemented by the inherited class but also some
methods that can be overwritten.
This patch also provides an example of a Scripted Process with the
`MyScriptedProcess` class.
rdar://65508855
Differential Revision: https://reviews.llvm.org/D95712
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
This patch adds a ScriptedProcess interface to the ScriptInterpreter and
more specifically, to the ScriptInterpreterPython.
This interface will be used in the C++ `ScriptProcess` Process Plugin to
call the script methods.
At the moment, not all methods are implemented, they will upstreamed in
upcoming patches.
This patch also adds helper methods to the ScriptInterpreter to
convert `SBAPI` Types (SBData & SBError) to `lldb_private` types
(DataExtractor & Status).
rdar://65508855
Differential Revision: https://reviews.llvm.org/D95711
Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
The Python code generated by SWIG is compatible with both Python 2 and
Python 3. The -py3 option enables Python 2 incompatible features such as
function annotations and abstract base classes.
Differential revision: https://reviews.llvm.org/D96096
Enums and constants are currently missing in the new LLDB Python API docs.
In theory we could just let them be autogenerated like the SB API classes, but sadly the generated documentation
suffers from a bunch of problems. Most of these problems come from the way SWIG is representing enums, which is
done by translating every single enum case into its own constant. This has a bunch of nasty effects:
* Because SWIG throws away the enum types, we can't actually reference the enum type itself in the API. Also because automodapi is impossible to script, this can't be fixed in post (at least without running like sed over the output files).
* The lack of enum types also causes that every enum *case* has its own full doc page. Having a full doc page that just shows a single enum case is pointless and it really slows down sphinx.
* There is no SWIG code for the enums, so there is also no place to write documentation strings for them. Also there is no support for copying the doxygen strings (which would be in the wrong format, but better than nothing) for enums (let alone our defines), so we can't really document all this code.
* Because the enum cases are just forwards to the native lldb module (which we mock), automodapi actually takes the `Mock` docstrings and adds it to every single enum case.
I don't see any way to solve this via automodapi or SWIG. The most reasonable way to solve this is IMHO to write a simple Clang tool
that just parses our enum/constant headers and emits an *.rst file that we check in. This way we can do all the LLDB-specific enum case and constant
grouping that we need to make a readable documentation page.
As we're without any real documentation until I get around to write that tool, I wrote a doc page for the enums/constants as a stop gap measure.
Most of this is done by just grepping our enum header and then manually cleaning up all the artifacts and copying the few doc strings we have.
Reviewed By: JDevlieghere
Differential Revision: https://reviews.llvm.org/D94959
The first line of the doc string ends up on the SB API class summary at
the root page of the Python API web page of LLDB. Currently many of the
descriptions are missing or are several lines which makes the table really
hard to read.
This just adds the missing docstrings where possible and fixes the formatting
where necessary.
This translates most of the old ASCII art in our documentation to the
equivalent in restructured text (which the new version of the LLDB docs
is using).
This reverts commit f775fe5964.
I fixed a return type error in the original patch that was causing a test failure.
Also added a REQUIRES: python to the shell test so we'll skip this for
people who build lldb w/o Python.
Also added another test for the error printing.
`struct Py_buffer_RAII` definition uses explicit deleted functions which are not supported by SWIG 2 (only 3).
To get around this I moved this struct to an .h file that is included to avoid being parsed by swig.
Reviewed By: lawrence_danna
Differential Revision: https://reviews.llvm.org/D86381
LLVM install component targets needs to be in the form of: install-{target}[-stripped]
I tested with:
```
cmake ... -DLLVM_ENABLE_PROJECTS="clang;lldb" -DLLVM_DISTRIBUTION_COMPONENTS="lldb;liblldb;lldb-python-scripts;" ...
DESTDIR=... ninja install-distribution
```
@JDevlieghere `finish_swig_python_scripts` is a really weird name for a distribution component, any reason that it has to be this way?
Differential Revision: https://reviews.llvm.org/D86235
This patch is a big sed to rename the following variables:
s/PYTHON_LIBRARIES/Python3_LIBRARIES/g
s/PYTHON_INCLUDE_DIRS/Python3_INCLUDE_DIRS/g
s/PYTHON_EXECUTABLE/Python3_EXECUTABLE/g
s/PYTHON_RPATH/Python3_RPATH/g
I've also renamed the CMake module to better express its purpose and for
consistency with FindLuaAndSwig.
Differential revision: https://reviews.llvm.org/D85976
Med Ismail Bennani