Commit Graph

524 Commits

Author SHA1 Message Date
Tamas Berghammer ccb367609b Add a new task pool class to LLDB
The purpose of the class is to make it easy to execute tasks in parallel

Basic design goals:
* Have a very lightweight and easy to use interface where a list of
  lambdas can be executed in parallel
* Use a global thread pool to limit the number of threads used
  (std::async don't do it on Linux) and to eliminate the thread creation
  overhead
* Destroy the thread currently not in use to avoid the confusion caused
  by them during debugging LLDB

Possible future improvements:
* Possibility to cancel already added, but not yet started tasks
* Parallel for_each implementation
* Optimizations in the thread creation destroyation code

Differential revision: http://reviews.llvm.org/D13727

llvm-svn: 250820
2015-10-20 12:42:05 +00:00
Zachary Turner 7d6d218e12 Convert SWIG typemap string operations to PythonObjects.
llvm-svn: 250530
2015-10-16 17:51:49 +00:00
Oleksiy Vyalov e98628cecb Split Socket class into Tcp/Udp/DomainSocket subclasses.
http://reviews.llvm.org/D13754

llvm-svn: 250474
2015-10-15 23:54:09 +00:00
Zachary Turner 9c40264fda Introduce a `PythonFile` object, and use it everywhere.
Python file handling got an overhaul in Python 3, and it affects
the way we have to interact with files.  Notably:

1) `PyFile_FromFile` no longer exists, and instead we have to use
   `PyFile_FromFd`.  This means having a way to get an fd from
   a FILE*.  For this we reuse the lldb_private::File class to
   convert between FILE*s and fds, since there are some subtleties
   regarding ownership rules when FILE*s and fds refer to the same
   file.
2) PyFile is no longer a builtin type, so there is no such thing as
   `PyFile_Check`.  Instead, files in Python 3 are just instances
   of `io.IOBase`.  So the logic for checking if something is a file
   in Python 3 is to check if it is a subclass of that module.

Additionally, some unit tests are added to verify that `PythonFile`
works as expected on Python 2 and Python 3, and
`ScriptInterpreterPython` is updated to use `PythonFile` instead of
manual calls to the various `PyFile_XXX` methods.

llvm-svn: 250444
2015-10-15 19:35:48 +00:00
Zachary Turner 18426935ac Get Python unit tests working with Python 3.
There were a couple of issues related to string handling that
needed to be fixed.  In particular, we cannot get away with
converting `PyUnicode` objects to `PyBytes` objects and storing
the `PyBytes` regardless of Python version.  Instead we have to
store a `PyUnicode` on Python 3 and a `PyString` on Python 2.

The reason for this is that if you call `PyObject_Str` on a
`PyBytes` in Python 3, it will return you a string that actually
contains the string value wrappedin the characters b''.  So if we
create a `PythonString` with the value "test", and we call Str()
on it, we will get back the string "b'test'", which breaks string
equality.  The only way to fix this is to store a native
`PyUnicode` object under Python 3.

With this CL, ScriptInterpreterPythonTests unit tests pass 100%
under Python 2 and Python 3.

llvm-svn: 250327
2015-10-14 21:06:13 +00:00
Zachary Turner f8b22f8fea Fix ref counting of Python objects.
PythonObjects were being incorrectly ref-counted. This problem was
pervasive throughout the codebase, leading to an unknown number of memory
leaks and potentially use-after-free.

The issue stems from the fact that Python native methods can either return
"borrowed" references or "owned" references. For the former category, you
*must* incref it prior to decrefing it. And for the latter category, you
should not incref it before decrefing it. This is mostly an issue when a
Python C API method returns a `PyObject` to you, but it can also happen with
a method accepts a `PyObject`. Notably, this happens in `PyList_SetItem`,
which is documented to "steal" the reference that you give it. So if you
pass something to `PyList_SetItem`, you cannot hold onto it unless you
incref it first. But since this is one of only two exceptions in the
entire API, it's confusing and difficult to remember.

Our `PythonObject` class was indiscriminantely increfing every object it
received, which means that if you passed it an owned reference, you now
have a dangling reference since owned references should not be increfed.
We were doing this in quite a few places.

There was also a fair amount of manual increfing and decrefing prevalent
throughout the codebase, which is easy to get wrong.

This patch solves the problem by making any construction of a
`PythonObject` from a `PyObject` take a flag which indicates whether it is
an owned reference or a borrowed reference. There is no way to construct a
`PythonObject` without this flag, and it does not offer a default value,
forcing the user to make an explicit decision every time.

All manual uses of `PyObject` have been cleaned up throughout the codebase
and replaced with `PythonObject` in order to make RAII the predominant
pattern when dealing with native Python objects.

Differential Revision: http://reviews.llvm.org/D13617
Reviewed By: Greg Clayton

llvm-svn: 250195
2015-10-13 18:16:15 +00:00
Zachary Turner 22c8efcd34 Port native Python-API to 3.x
With this change, liblldb is 95% of the way towards being able
to work under both Python 2.x and Python 3.x.  This should
introduce no functional change for Python 2.x, but for Python
3.x there are some important changes.  Primarily, these are:

1) PyString doesn't exist in Python 3.  Everything is a PyUnicode.
   To account for this, PythonString now stores a PyBytes instead
   of a PyString.  In Python 2, this is equivalent to a PyUnicode,
   and in Python 3, we do a conversion from PyUnicode to PyBytes
   and store the PyBytes.
2) PyInt doesn't exist in Python 3.  Everything is a PyLong.  To
   account for this, PythonInteger stores a PyLong instead of a
   PyInt.  In Python 2.x, this requires doing a conversion to
   PyLong when creating a PythonInteger from a PyInt.  In 3.x,
   there is no PyInt anyway, so we can assume everything is a
   PyLong.
3) PyFile_FromFile doesn't exist in Python 3.  Instead there is a
   PyFile_FromFd.  This is not addressed in this patch because it
   will require quite a large change to plumb fd's all the way
   through the system into the ScriptInterpreter.  This is the only
   remaining piece of the puzzle to get LLDB supporting Python 3.x.

Being able to run the test suite is not addressed in this patch.
After the extension module can compile and you can enter an embedded
3.x interpreter, the test suite will be addressed in a followup.

llvm-svn: 249886
2015-10-09 19:45:41 +00:00
Bruce Mitchener eb284e4667 Fix segmentation fault in lldb_private::Symbols::LocateExecutableSymbolFile()
Summary:
When `module_spec.GetFileSpec().GetDirectory().AsCString()` returned a `nullptr` this line caused a segmentation fault:

`std::string module_directory = module_spec.GetFileSpec().GetDirectory().AsCString()`

Some context:
I was remote debugging an executable built with Clang in an Ubuntu VM on my Windows machine using lldb-mi. I copied the executable and nothing else from the Ubuntu VM to the Windows machine.

Then started lldb-server in the Ubuntu VM:

```
./bin/lldb-server gdbserver *:8888 -- /home/enlight/Projects/dbgmits/build/Debug/data_tests_target
```

And ran `lldb-mi --interpreter` on Windows with the following commands:

```
-file-exec-and-symbols C:\Projects\data_tests_target
-target-select remote 192.168.56.101:8888
-exec-continue

```

After which the segmentation fault occurred at the aforementioned line. Inside this method `module_spec.GetFileSpec()` returns an empty `FileSpec` (no dir, no filename), while `module_spec.GetSymbolFileSpec().GetFilename()` returns `"libc-2.19.so"`.

Patch thanks to Vadim Macagon.

Reviewers: brucem, zturner, clayborg

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D13201

llvm-svn: 249387
2015-10-06 10:17:34 +00:00
Oleksiy Vyalov 549718563d Make UriParser to support [$HOSTNAME] notation.
http://reviews.llvm.org/D12025

llvm-svn: 245639
2015-08-20 23:09:34 +00:00
Zachary Turner 2c1f46dcc6 Convert the ScriptInterpreter system to a plugin-based one.
Previously embedded interpreters were handled as ad-hoc source
files compiled into source/Interpreter.  This made it hard to
disable a specific interpreter, or to add support for other
interpreters and allow the developer to choose which interpreter(s)
were enabled for a particular build.

This patch converts script interpreters over to a plugin-based system.
Script interpreters now live in source/Plugins/ScriptInterpreter, and
the canonical LLDB interpreter, ScriptInterpreterPython, is moved there
as well.

Any new code interfacing with the Python C API must live in this location
from here on out.  Additionally, generic code should never need to
reference or make assumptions about the presence of a specific interpreter
going forward.

Differential Revision: http://reviews.llvm.org/D11431
Reviewed By: Greg Clayton

llvm-svn: 243681
2015-07-30 20:28:07 +00:00
Bruce Mitchener 28529f607c Fix StringExtractor.h issues.
Summary:
Fix StringExtractor.h issues.

* source/Plugins/Process/MacOSX-Kernel/ProcessKDP.cpp
  (#include "Utility/StringExtractor.h): Not needed, this is already
    included by ProcessKDP.h

* unittests/Utility/StringExtractorTest.cpp
  (#include "Utility/StringExtractor.h): Update include path to the
    new location.

Reviewers: labath, clayborg

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D10995

llvm-svn: 241596
2015-07-07 15:19:03 +00:00
Zachary Turner c48ec9d0ff Including <thread> with MSVC is buggy, use a workaround here.
llvm-svn: 238469
2015-05-28 19:56:43 +00:00
Pavel Labath c076559a5b [NativeProcessLinux] fold ThreadStateCoordinator into NPL
Summary:
Since all TSC operations are now executed synchronously, TSC has become a little more than a
messenger between different parts of NativeProcessLinux. Therefore, the reason for its existance
has disappeared.

This commit moves the contents of the TSC into the NPL class. This will enable us to remove all
the boilerplate code in NPL (as it stands now, this is most of the class), which I plan to do in
subsequent commits.

Unfortunately, this also means we will lose the unit tests for the TSC. However, since the size
of the TSC has diminished, the unit tests were not testing much at this point anyway, so it's not
a big loss.

No functional change.

Test Plan: All tests continue to pass.

Reviewers: vharron, chaoren

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D9296

llvm-svn: 236587
2015-05-06 10:46:34 +00:00
Pavel Labath 45f5cb31dc [NativeProcessLinux] Get rid of the thread state coordinator thread
Summary:
This change removes the thread state coordinator thread by making all the operations it was
performing synchronous. In order to prevent deadlock, NativeProcessLinux must now always call
m_monitor->DoOperation with the m_threads_mutex released. This is needed because HandleWait
callbacks lock the mutex (which means the monitor thread will block waiting on whoever holds the
lock). If the other thread now requests a monitor operation, it will wait for the monitor thread
do process it, creating a deadlock.

To preserve this invariant I have introduced two new Monitor commands: "begin operation block"
and "end operation block". They begin command blocks the monitor from processing waitpid
events until the corresponding end command, thereby assuring the monitor does not attempt to
acquire the mutex.

Test Plan: Run the test suite locally, verify no tests fail.

Reviewers: vharron, chaoren

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D9227

llvm-svn: 236501
2015-05-05 15:05:50 +00:00
Pavel Labath 980b92a8e6 Add missing libraries to unittest link
Summary:
Currently, linking of the unittests fails on linux because it is missing a bunch of symbols from
libedit, curses, etc. This fixes the build by adding the correct dependencies.

Test Plan: Linking works, unit tests run.

Reviewers: zturner

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D9244

llvm-svn: 235853
2015-04-27 09:26:03 +00:00
David Blaikie 63097d3212 Fix the Linux build.
llvm-svn: 233990
2015-04-03 01:12:52 +00:00
Zachary Turner 799770c03a Fix linking of unit tests via CMake on Windows.
A previous attempt to make the unit tests link properly on
Linux broke it for Windows.  This patch fixes it for both platforms.

llvm-svn: 232648
2015-03-18 16:56:24 +00:00
Zachary Turner 85bc48ca74 Convert CRLF to LF.
I accidentally let some Windows line endings slip in.  This is a
good reminder for me to use core.eol=lf.

llvm-svn: 232560
2015-03-17 22:51:21 +00:00
David Blaikie 3dfb86b641 Fix the clang -Werror build & make the unit tests link under Linux
The order of libraries passed to the linker didn't work under linux (you
need the llvm libraries first, then the lldb libraries). I modelled this
after clang's setup here. Seemed simple enough to just be consistent.

llvm-svn: 232461
2015-03-17 03:32:21 +00:00
Zachary Turner 073951f28b Fix a bug related to arg escaping, and add unit tests.
A recent refactor had introduced a bug where if you escaped a
character, the rest of the string would get processed incorrectly.

This patch fixes that bug and adds some unit tests for Args.

llvm-svn: 232288
2015-03-14 23:39:42 +00:00
Zachary Turner eeba1a896a [CMake] Make the unittests link against everything...
Sigh.  There's really not a good alternative until we decouple
python from lldb better.  The only way the build works right now
is by having every executable link against every LLDB library.
This causes implicit transitive link dependencies on the union
of everything that LLDB brings in.  Which means that if all we
want is one header file from interpreter, we have to bring in
everything, including everything that everything depends on,
which means python.

There's outstanding efforts to address this, but it's not yet
complete.  So until then, this is all we can do.

llvm-svn: 232287
2015-03-14 23:39:28 +00:00
Zachary Turner 2c04f79b01 [gtest] Fix gtest failures on Windows.
On Windows, you need to call WSAStartup() before making any socket
calls, and WSACleanup() before you shutdown.  This wasn't being
done, so all of the socket tests were failing.  This fixes
that, which brings the unit test suite to a fully working state
on Windows.

llvm-svn: 232247
2015-03-14 04:19:32 +00:00
Zachary Turner a6ad7a542b [CMake] Fix unit test build on Linux.
A filename had been misspelled in the CMake.

llvm-svn: 232218
2015-03-13 21:33:34 +00:00
Zachary Turner 719ec93a74 Rework the gtest directory structure.
This makes the directory structure mirror the canonical LLVM
directory structure for a gtest suite.

Additionally, this patch deletes the xcode project.  Nobody
is currently depending on this, and it would be better to have
gtest unit tests be hand-maintained in the Xcode workspace
rather than using this python test runner.  Patches to that
effect will be submitted as followups.

llvm-svn: 232211
2015-03-13 20:54:57 +00:00