Commit Graph

391 Commits

Author SHA1 Message Date
Panu Matilainen 2986e7659d Remove redundant max_macro_depth initialization
Oldies but goodies: the max depth is already initialized in declaration
to _MAX_MACRO_DEPTH which has been the same 16 since late nineties. This
wasn't always the case of course...
2017-08-18 17:12:47 +03:00
Pavlina Moravcova Varekova 94e8cd6058 Make macro %{error:...} cause an actual failure 2017-08-17 16:16:23 +03:00
Pavlina Moravcova Varekova a3b153eb47 Unified messages printed by %{echo:...}, %{warn:...}, and %{error:...}
Send all through rpmlog(), unify formatting - previously echo and
warn were missing the trailing newline. The other change is that
echo now goes to stdout.
2017-08-17 16:15:39 +03:00
Panu Matilainen a3c1f733e0 Revert "Support quoting of macro arguments (#222)"
This simply breaks too many things - whole macro ecosystems exist based
on the assumption that quotes in arguments will pass to macros
untouched. Macro argument quoting support is necessary but it'll
need some entirely different approach that is either opt-in or
based on a different syntax altogether.

This reverts commit 47f0a899ef.
2017-08-14 14:28:10 +03:00
Jeff Johnson 3684424fe2 Add support for zstd compressed payload
v2 (Igor Gnatenko):
    * Switch off from RPM_CHECK_LIB
    * Reference zstd from rpm.pc
    * Link rpmio with zstd
  v3 (Florian Festi):
    * move changes to cvtfmode into separate patches
    * do not error out on wrong compression levels
    * ifdef out zstdio
    Closes: https://github.com/rpm-software-management/rpm/issues/256
    Closes: https://github.com/rpm-software-management/rpm/issues/297
    Signed-off-by: Igor Gnatenko <i.gnatenko.brain@gmail.com>
2017-08-09 17:41:03 +02:00
Pavlina Moravcova Varekova c4d41c30ce Shift macro scoping level values as they are defined
Some of scoping level values are defined in rpmmacros.h:
 #define RMIL_DEFAULT    -15
 #define RMIL_MACROFILES -13
...
 #define RMIL_GLOBAL     0

Before this patch RPM set these values decreased of 1 (for all macros).
Because of the (level - 1) in
    pushMacro(dst, me->name, me->opts, me->body, (level - 1), me->flags);
in function copyMacros scoping level for some CMDLINE macros was decreased of 1.
Because of the (level - 1) in
    pushMacro(mb->mc, n, o, b, (level - 1), ME_NONE);
in function doDefine scoping level values of all other macro types and
some CMDLINE macros was decreased of 1.
The patch shifts scoping level values back to the numbers defined
in rpmmacros.h.

This was spotted when investigating a regression introduced in
commit 237f35f16f where locally scoped
scoped macros did not get deleted on scope closing.
This also fixes the regression in question.

The first test in this patch tests automatic deleting of %defined macros.
The second test checks whether the behavior changed by
237f35f16f is still working correctly.

Thanks to Panu Matilainen for investigation and proposal of the main part
of the patch!
2017-08-09 14:36:15 +03:00
Pavlina Moravcova Varekova 6ca89685e3 Distinguish %define and %global in the illegal macro name error message 2017-08-09 13:31:18 +03:00
Pavlina Moravcova Varekova 47f0a899ef Support quoting of macro arguments (#222)
Single or double quotes can be used in macro arguments
as expected - they denote the begin and the end of the argument.

Example with 2-nd and 3-rd argument in double quotes:
./rpm --define "%foo() 1:%1 2:%2 3:%3" \
         --eval '%foo Next_argument_will_be_empty "" "Last argument"'
1:Next_argument_will_be_empty 2: 3:Last argument

Example with all arguments in single quotes,
without spaces between arguments:

./rpm --define "%foo() 1:%1 2:%2 3:%3" \
         --eval "%foo 'Next argument will be empty''''Last argument'"
1:Next argument will be empty 2: 3:Last argument
2017-08-09 13:02:54 +03:00
Per Øyvind Karlsen 7fe183d660 Add support for %{shrink:} (ported from rpm5.org)
Shrinks body by removing all leading and trailing whitespaces and
reducing intermediate whitespaces to a single space character.
2017-08-09 12:57:22 +03:00
Ville Skyttä 2901e265a0 Spelling fixes 2017-06-27 14:42:21 +02:00
Panu Matilainen 85953c0d72 Revert "Expand parameterized macro arguments one by one after splitting"
This essentially reverts commit 9ae7d1df31.
Leaving the testcase in place, just adjusting it to the expected
behavior where arguments that expand to empty are eaten away.

This actually matches what the shell does, so it follows the path of
least surprise in that sense - we emulate shell behavior with the
arguments etc already in many ways. What the shell has and rpm
has not is a way to pass those empty arguments by quoting. So
now we need a way to quote arguments - not just for passing empty
arguments but arguments containing whitespaces too.
2017-05-23 11:15:31 +03:00
Panu Matilainen 767d61ca3d Preserve %%-escapes when expanding parametric macro arguments (#217)
There needs to be a way to pass escaped macros names (and other strings
containging %%) as parametric macro arguments. Add an internal mode
to preserve %% when expanding the arguments. Fixes a regression introduced
in commit 5adc56897b.

Thanks to Michael Schroeder for pointing this out!
2017-05-18 13:47:21 +03:00
Panu Matilainen 9ae7d1df31 Expand parameterized macro arguments one by one after splitting
This makes makes the behavior much more sensible eg in case an
argument expands to an empty strings, such as:
    $ rpm --define '%foo() 1:%1 2:%2' --eval '%foo %nil bar'
After commit 5adc56897b rpm was
evaluating that to "1:bar 2:%2" which is hardly what you'd expect.

Thanks to Michael Schroeder for pointing this out! (#217)
2017-05-18 12:48:14 +03:00
Panu Matilainen 117b337fb2 Remember flags passed to rpmExpandMacros() in the expansion state buffer
Of course right now there are no implemented flags so this doesn't
actually do anything.
2017-05-18 11:19:13 +03:00
Panu Matilainen 5adc56897b Expand parametric macro arguments before processing (#127, RhBug:1397209)
This too is quite a fundamental change for macros: up to now, arguments
to parametric macros have not been expanded. It doesn't sound so bad
until you consider something like the case in RhBug:1397209:

        %global rev 133
        ...
        %setup %{?rev:-c}
        %autosetup %{?rev:-c}

One would expect %setup and %autosetup behave the same when you replace
one with the other, but no.  %setup gets "called" with -c, %autosetup
does not. Instead %autosetup receives a completely useless, literal
"%{?rev:-c}" as its argument.  That's just brain-meltingly non-sensical.

This certainly has the potential to break advanced macro constructs,
but OTOH what breaks might well be only written that way in order to
work around the former behavior.

There are some funny subtleties involved as the argument expansion
must occur in the callers scope, ie before we create any of the
automatic macros. For example, Fedora's font packages break if only
this or the macro scope visibility enforcement is applied but start
working again once both are present.
2017-03-07 14:26:45 +02:00
Panu Matilainen 307627993e Enforce visibility scoping for automatic macros
When a parametric macro "calls" another parametric macro with fewer
arguments than it received, the inner macro would see the %<n>
macros of the outer call which is an obvious scoping violation
and quirky behavior, making macro writing harder than it needs be.

Similar scoping issues exist for manually defined macros but those
have further complications (because of %undefine semantics)  that we
sheepishly avoid here by limiting the visibility enforcing to automatic
macros only (for now at least).
2017-03-07 14:06:55 +02:00
Panu Matilainen 237f35f16f Change macro scoping to be global / local to parametric macros
This changes the macro scoping rules quite fundamentally: macro
definitions are local to the parametric macro they were defined in,
and everything else is global. Among other things, this makes this
common spec idiom (RhBug:552944, RhBug:551971 etc) behave
deterministically because "foo" is placed into global scope:

        %{?!foo: %define foo bar}

In theory it's certainly possible that this breaks somebodys carefully
crafted advanced macros but it seems quite unlikely, considering how
broken the alleged block-scoping has been. OTOH for macros defined
within parametric macros, nothing actually changes as that scoping has
always been enforced by rpm. The non-global define tracking is also
no longer needed for emitting warnings, because the case where it
was needed simply no longer exists.

Note that macro recursion depth is a different thing and still needs
to be tracked separately.
2017-03-07 14:03:12 +02:00
Panu Matilainen e2f4347356 Consolidate all macro argument setup to grabArgs()
Since we actually setup all the same automatic macros whether there
are arguments or not, doing it centrally only makes sense. Shuffle
things around a bit in preparation for the next steps.
2017-03-07 13:58:56 +02:00
Panu Matilainen 3b1f4b0c6c Cosmetics: if, while and switch are followed by a space
The missing space style-error has been recently coming common enough
somebody might think that IS the expected style, its not. Some of these
are actually very old, but fix across the board for consistency..

Strictly white-space only change.
2017-02-27 17:41:37 +02:00
Peter Jones e6f346ead4 Don't nerf "rpmspec --eval '%trace' -P foo.spec" output with ellipses.
The whole point of %trace is that some macro is being evaluated in a
surprising way, and you want to know what it is.  This is often the
result of an unfortunately complex macro with difficult to see
intermediate results.

Shortening the output because it doesn't fit past 61 or so characters
does not help, and we have big screens these days.  Don't do it.

Signed-off-by: Peter Jones <pjones@redhat.com>
2017-02-24 13:23:16 +01:00
Peter Jones bf248badd3 Bounds check strings to print correctly in %trace mode.
In %trace mode, evaluating a macro which is undefined causes an invalid
read of 1 byte when searching for the end of the string:

trillian:~$ valgrind rpmspec --eval '%trace' --eval '%{?myUndefinedMacro}'
==21534== Memcheck, a memory error detector
==21534== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==21534== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==21534== Command: rpmspec --trace --eval %{?myUndefinedMacro}
==21534==

  1>   %{?myUndefinedMacro}^==21534== Invalid read of size 1
==21534==    at 0x55018D4: printMacro (macro.c:296)
==21534==    by 0x5502DFC: expandMacro (macro.c:1077)
==21534==    by 0x5503710: doExpandMacros (macro.c:1280)
==21534==    by 0x5504AB6: rpmExpand (macro.c:1629)
==21534==    by 0x508F59A: rpmcliAllArgCallback (poptALL.c:120)
==21534==    by 0x6DAF71D: invokeCallbacksOPTION (popt.c:156)
==21534==    by 0x6DAF75B: invokeCallbacksOPTION (popt.c:139)
==21534==    by 0x6DB1428: poptGetNextOpt (popt.c:1515)
==21534==    by 0x508F912: rpmcliInit (poptALL.c:302)
==21534==    by 0x1095B2: main (rpmspec.c:63)
==21534==  Address 0x8a010f3 is 0 bytes after a block of size 19 alloc'd
==21534==    at 0x4C2DB9D: malloc (vg_replace_malloc.c:299)
==21534==    by 0x5507C17: rmalloc (rpmmalloc.c:44)
==21534==    by 0x5502788: expandMacro (macro.c:927)
==21534==    by 0x5503710: doExpandMacros (macro.c:1280)
==21534==    by 0x5504AB6: rpmExpand (macro.c:1629)
==21534==    by 0x508F59A: rpmcliAllArgCallback (poptALL.c:120)
==21534==    by 0x6DAF71D: invokeCallbacksOPTION (popt.c:156)
==21534==    by 0x6DAF75B: invokeCallbacksOPTION (popt.c:139)
==21534==    by 0x6DB1428: poptGetNextOpt (popt.c:1515)
==21534==    by 0x508F912: rpmcliInit (poptALL.c:302)
==21534==    by 0x1095B2: main (rpmspec.c:63)
==21534==

  1>   %{?_transaction_color}^
  1>   %{?_prefer_color}^
  1>   %{_netsharedpath}^
  1>   %{_install_langs}^
==21534==
==21534== HEAP SUMMARY:
==21534==     in use at exit: 7,183 bytes in 71 blocks
==21534==   total heap usage: 7,811 allocs, 7,740 frees, 3,500,361 bytes allocated
==21534==
==21534== LEAK SUMMARY:
==21534==    definitely lost: 19 bytes in 1 blocks
==21534==    indirectly lost: 0 bytes in 0 blocks
==21534==      possibly lost: 0 bytes in 0 blocks
==21534==    still reachable: 7,164 bytes in 70 blocks
==21534==         suppressed: 0 bytes in 0 blocks
==21534== Rerun with --leak-check=full to see details of leaked memory
==21534==
==21534== For counts of detected and suppressed errors, rerun with: -v
==21534== ERROR SUMMARY: 5 errors from 1 contexts (suppressed: 0 from 0)
trillian:~$

This can easily be avoided by checking the first byte as well as the
second for our sentinal value (NUL).

Signed-off-by: Peter Jones <pjones@redhat.com>
2017-02-24 13:23:16 +01:00
Panu Matilainen 1767bc4fd8 Preserve macro scoping level on re-entry from Lua
When a %{lua:...} macro calls rpm.expand(), it re-enters the macro
environment at depth zero regardless of what the "calling" nesting
level was. This could lead to wrong macros getting mopped out on
complex, nested lua + native macro calls.

Store the depth in global macro context and pick up the initial value
from there to keep the nesting level increasing on when entering %{lua:...}
space.
2017-01-20 15:23:01 +02:00
Panu Matilainen f95825ef1b Add return code to rpmPushMacro() and rpmPopMacro()
These haven't been in any public release yet so there's time to
fiddle with stuff like this. Both always succeed currently so no
change in that respect but it's better to have room for expansion...
2017-01-19 16:03:15 +02:00
Panu Matilainen 31bc44c5a9 Actually return errors from macro definition 2017-01-19 14:24:26 +02:00
Panu Matilainen bd942d5fee Raise actual error on unknown option to parametric macro 2017-01-19 14:18:29 +02:00
Panu Matilainen 02dea59e3a Flag real errors from macro define and undefine errors
This WILL cause behavior changes in cases where there have been
illegal macro names etc which have previously been more or less
ignored with an logged error, but now they will actually abort.
2017-01-19 14:15:38 +02:00
Panu Matilainen 8f86427d8a Pass macro buf, not context, to doUndefine()
No functional changes, just for symmetry with doDefine()
2017-01-19 14:13:15 +02:00
Panu Matilainen 7d1ceefe9e Store errors during macro expansion centrally
With direct rc assignments there's a risk that a previously
flagged error is overwritten by a success, only flagging errors
specifically avoids that. It also makes things simpler in some ways.

This probably changes *some* behavior wrt errors, in that errors
might actually get counted as such.
2017-01-19 14:08:28 +02:00
Panu Matilainen b298e93686 Arrange single point of exit in expandMacro() 2017-01-19 13:41:42 +02:00
Panu Matilainen 1b4744e1c6 Refactor %{lua:...} macro execution to a helper function
expandMacro() is big enough without this kind of easily separable
material...
2017-01-19 12:09:10 +02:00
Panu Matilainen e3e99e929a Reindent expandMacro() to common rpm style
This huge and scary looking diff is strictly whitespace-only change.
2017-01-19 11:28:40 +02:00
Panu Matilainen 5517d26061 Rename addMacro() and delMacro() to rpmPushMacro() and rpmPopMacro()
These are not deprecated at all no matter what the header has been
saying for the past 15+ years, they're used by rpm itself all over
the place as rpmDefineMacro() serves a slightly different purpose
and there's no rpmUndefineMacro() anyway.

Lets make 'em into proper citizens and move them into rpm namespace,
and while at it, call the operations push and pop since that's much
closer to what actually happens.

Finally, add simple wrapper macros to keep external code compilable
while getting the non-namespaced stuff out of ABI.
2016-10-24 13:09:38 +03:00
Panu Matilainen 34e839b845 Eliminate expandMacros() from the API + ABI
This allegedly been deprecated for no less than sixteen years,
but until very recently it's been in use by rpm itself and only
now with rpmExpandMacros() its possible to eliminate this one.
All sane external callers have been using rpmExpand() anyway
since expandMacros() is "buffer limited".
2016-10-24 12:51:13 +03:00
Lubos Kardos ddf9ec7bef rpmExpandMacros() is modified to be able to return more return codes
Now rpmExpandMacros() returns integer as a return value. Negative return
value implies failure. Return values can be more utilized in the future.
E. g. return value could specify how many macros were expanded in given
string.
2015-11-26 15:54:27 +01:00
Tom Hughes aee8446eb4 Rename expandMacrosU to rpmExpandMacros
Address review issues from #32 by renaming the function and
cleaning up the comment and parameter list.
2015-11-26 15:54:27 +01:00
Tom Hughes 61838b0fda Remove size limit when expanding macros
This removes a seemingly undocumented, and not even well defined, limit
on the size of a macro expansion when parsing a spec file.

[lkardos@redhat.com: created new funtion expandMacrosU() (Unlimited)
instead of modifying expandMacros() in order not to change API/ABI]

Signed-off-by: Lubos Kardos <lkardos@redhat.com>
2015-11-19 12:42:59 +01:00
Lubos Kardos 54f24ec548 Fix reading a memory right after the end of an allocated area.
The problem evinced itself when somebody tried to use the macro
expansion on the string "%!". The problem was revealed by compiling
with "--fsanitize=memory" (rhbz:#1260248).
2015-09-21 11:34:42 +02:00
Florian Festi b151b5297b Fix doxygen warnings
Make shure parameter descriptions in doc strings match the functions
2015-02-17 15:31:22 +01:00
Panu Matilainen ace7fe9d58 Enable the unused macro warning again, now without performance penalty
- Track non-global macro definitions in a helper variable, this
  eliminates the need for costly macro table walking at end of each
  expansion cycle as we only bother doing it if non-global macros
  were defined during a given expansion cycle.
2014-09-02 11:50:41 +03:00
Panu Matilainen d4ab1d82d7 Macro-expand %{load:...} argument to make the thing more useful... 2014-07-01 12:05:51 +03:00
Panu Matilainen 817959609b Handle line continuation in grabArgs() (related to RhBug:1045723)
- Commit 1bdcd05008 to fix RhBug:1045723
  broke some funky java macros in Fedora which include line continuation
  in the argument (comments 6-7 in the bug). That it ever worked seems
  far more like luck than by design but since this seems to fix it...
2014-06-25 11:32:24 +03:00
Panu Matilainen eb62542695 Fix ancient buffer overflow on unterminated macro options
- Test for terminating ')' existence before copying, otherwise we'll
  end up walking over the edge of the world.
- Return address from doDefine() on error will likely differ after
  this, whether that actually affects anything remains to be seen...
2014-06-11 15:40:00 +03:00
Panu Matilainen 4089316531 Fix ancient buffer overflows on macro define and undefine (RhBug:1087000)
- Both doDefine() and doUndefine() assumed the macro string would
  always fit into MACROBUFSIZ, which of course is true for any
  normal use but we cant make such assumptions.
- In the case of %define/%global there are various other overrun-issues
  that need further changes to fix.
2014-06-09 14:35:19 +03:00
Panu Matilainen f07490f5dc Only enable the unused macro warning on errors and when tracing
- In the current macro implementation the check is simply far too
  expensive to leave on always. Its useful though, so enable it
  when tracing macro expansion, in which case you're probably
  troubleshooting some macro-issues...
2014-03-26 11:43:41 +02:00
Panu Matilainen b19348342b And finally, actually enable the unused macro warning.
- Reword the message to something hopefully little more understandable,
  change from error to a warning (only matters for output)
- Check unused macros at end of every scope, but unlike with parametrized
  macros, dont actually delete
- Only whine once per unused macro due to the above to avoid spurious
  output due to above
- This catches the common error in specs where %define is used in a scope
  unintentionally, eg "%{!?foo: %define foo 1}" where the just defined
  macro should actually fall out of scope upon the closing }, but
  only gets erased if any parametrized macro is "called" and causes
  insane behavior such as described in RhBug:552944. So at least we'll
  warn on these situations.
2014-02-28 16:45:33 +02:00
Panu Matilainen f9a6855faa Consider macro used if we've so much as looked it up 2014-02-28 16:38:56 +02:00
Panu Matilainen e65e27ab25 Make macro deletion in freeArgs() optional
- Not used yet, but using this for issuing warnings only
2014-02-28 16:12:33 +02:00
Panu Matilainen ae58c96248 Move macro usage tracking to flags
- We dont really care how many times a macro has been used, just
  whether it has been used or not...
2014-02-28 16:10:14 +02:00
Panu Matilainen fae7325f8d Add + use a flag for identifying automatic macros
- In practise doesn't change anything visible as the "macro not used"
  message is disabled, but tracking a flag bit is saner than
  strlen() + strchr() + bunch of other stuff which isn't even correct:
  prior to this %## would've triggered "unused" errors which is plain
  wrong, and complaining about unused %1 %2 ... isn't really right
  either.
2014-02-28 15:57:09 +02:00
Panu Matilainen 647f0d488f Revert stricter macro substitution syntax entirely, for now
- Revert back to pre commit c22d5b1299
  state wrt macro substitution. The commit does have fixes we want
  but it also breaks valid cases which is not okay. We really need
  a far more thorough macro test suite before these kind of changes,
  and also need to do the changes in more controlled pieces.
2014-02-25 16:17:37 +02:00
Panu Matilainen df686992d7 Perform as-is substitution for %{name} form too, again
- Commit c22d5b1299 changed the parsing
  to require a valid name in %{name} macro, but this outlaws existing
  uses such as %{foo_%{bar}_something}. Relaxing the %{name} form
  to use as-is substitution as well allows these to work again.
2014-02-20 10:42:09 +02:00
Panu Matilainen 1bdcd05008 Dont eat newlines on parametrized macro invocations (RhBug:1045723)
- Makes the testcase from commit f082b5baa4
  succeed. While the old behavior is non-sensical and most likely entirely
  unintentional, we're changing a very long-standing behavior here (tested
  back to rpm 4.4.x and almost certainly much much older than that) so
  its entirely possible people are actually relying on the old
  behavior. Lets see what breaks...
2014-02-19 14:24:54 +02:00
Panu Matilainen 784dfb4219 Port %{load:<path>} macro from rpm5.org
- Allow loading custom macro files from eg specs. This is quite different
  from %include which inlines arbitrary content into a spec, but which
  cannot be used for including macro files as their syntax is entirely
  different. Both have their uses.
- rpm5.org also supports %load within macro files, we dont as I find the
  imperative %load very alien in what's otherwise an entirely declarative
  file "format"
2014-01-30 14:57:52 +02:00
Panu Matilainen ae57958971 Fix ages old segfault on trying to expand %{lua} (RhBug:972646)
- Using the already calculated macro arg + arglen for copying the
  buffer instead of buggy local variant helps... This is as old as
  the embedded lua interpreter in rpm, close to a decade.
- Besides fixing the segfault, this actually makes it behave like
  other built-ins, evaluating to empty string when an empty arguments
  or no arguments are passed.
2013-06-11 11:33:53 +03:00
Panu Matilainen 99b40fe5ab Dont try to expand zero-length arguments to built-in macros
- Previously various built-in macros without an actual argument,
  eg %{basename:} would evaluate to "}" which makes no sense
  whatsoever. With this, they evaluate to an empty string similarly
  to when no argument is passed, eg %{basename}. Arguably these
  should emit an error/warning instead but for now...
2013-06-11 10:46:50 +03:00
Panu Matilainen f173f747cd Revert "Fix segfault trying to expand %{lua} (RhBug:972646)"
- This reverts commit 43a34e1554,
  contrary to what the comment said the is NOT how other built-ins
  behave, they evaluate to an empty string without an argument.
  Better fix needed...
2013-06-10 15:31:01 +03:00
Panu Matilainen 43a34e1554 Fix segfault trying to expand %{lua} (RhBug:972646)
- %{lua:...} is used for invoking the embedded Lua interpreter, in
  which case the script must be passed as the macro argument. If
  no argument is passed, let it fall through to normal macro expansion.
  This might not be the most sensible behavior possible but at least
  its in line with what currently happens with other similar built-in
  macros.
2013-06-10 14:44:55 +03:00
Panu Matilainen 71e2d38e3d Make macros thread-safe ... sort of
- Add the actual locking into macro context acquire + release. We're
  using a mutex instead of rwlock because currently any macro can
  involve defines and undefines and we dont know it beforehand. So
  we just use a bigger hammer...
- The macro engine internals shouldn't need recursive mutexes but
  unfortunately Lua macro bindings, which are limited to the
  lock-on-entry interfaces can and do get called recursively
  from macros so currently there's not much choice but to use
  recursive mutex for the macro contexts. What makes it even uglier
  is that there's no portable static initializer for recursive mutex
  so we need to add yet another pthread-construct to safely dynamically
  initialize the mutexes :(
- Of course this doesn't make bunch of threads simultaneously messing
  with macros behave sane: if one thread tries to load new macros
  and the other one frees them, they shouldn't crash but the
  results are unlikely to be what the caller intended. The purpose
  here is just to allow the occasional rpmExpand() and such to
  complete without crashing and burning when multiple threads are
  doing stuff like trying to read packages from disk.
2013-05-22 15:35:29 +03:00
Panu Matilainen 09499d994b Add and use internal (lockless) function for copying macro contexts
- rpmLoadMacros() is a dumb name for what it does: it copies macros
  from one context to another. The only actual use within rpm is
  to copy back the cli macros to global context, but make the
  internal helper more flexible by allowing copying to any context.
- rpmLoadMacros() is mostly just a dumb wrapper around copyMacros()
  to grab locks and guard against copying global context to itself.
  Adjust rpmInitMacros() to use copyMacros() as it already has a
  lock on the global table, it just needs a lock on the cli contexts
  as well.
2013-05-22 11:43:40 +03:00
Panu Matilainen 62d36cce61 Add and use internal (lockless) version of rpmLoadMacroFile()
- rpmInitMacros() already grabs the (theoretical) lock on entry
  so we shouldn't try to grab it again, use the new lockless
  loadMacroFile() version for the purpose.
- rpmLoadMacroFile() is now just a simple lock-wrapper around
  loadMacroFile()
2013-05-22 11:27:29 +03:00
Panu Matilainen bfda1b6013 Add and use internal (lockless) version of rpmDefineMacro()
- rpmLoadMacroFile() already grabs the (theoretical) lock on entry
  so we shouldn't try to grab it again, use the new lockless
  defineMacro() version for the purpose.
- rpmDefineMacro() is now just a simple lock-wrapper around defineMacro()
2013-05-22 11:22:08 +03:00
Panu Matilainen 1036601e3e Add and use dummy macro context lock acquire + release functions
- Add and use dummy macro context lock acquire + release functions,
  ATM these only centralize the NULL -> rpmGlobalMacroContext translation.
- Eliminate the NULL context translation from internal interfaces,
  ensure its always done at the outer levels already.
- At least rpmLoadMacros() and rpmLoadMacroFile() are fishy wrt
  theoretical locking at this point and will require further internal
  helpers...
2013-05-22 11:10:15 +03:00
Panu Matilainen acead93927 Group exported functions to end of macro.c, no functional changes
- Having the external interfaces together makes it easier to keep
  tracking of locks and all..
2013-05-22 10:29:19 +03:00
Panu Matilainen b823075695 Separate internal vs external macro add and delete functions
- Rename addMacro() and delMacro() to pushMacro() and popMacro()
  respectively, change all internal callers to use the push/pop
  variants. Make addMacro() and delMacro() just dummy wrappers
  for the internal variants.
- No functional changes, just paving way for adding thread-safety
  mutex: we'll be locking at the external API boundary to keep
  things simple, avoid nested locking etc.
2013-05-22 10:18:42 +03:00
Panu Matilainen c012a76985 Optimize expandMacro() a bit, based on patch by Alexey Tourbin
- Avoid the expensive calloc() when creating a macro expansion buffer,
  malloc() suffices just as well as long as we initialize the first
  byte in the buffer. This is easily visible on wall-clock times.
- Avoid recalculating source string length: when caller doesn't
  pass slen, we always calculate the string length for creating
  a terminated copy, no need to redo it...
- Add a warning comment about the seemingly obvious optimization
  breaking macro undefining itself, this is starting to look like
  a recurring theme :)
2013-04-16 11:07:03 +03:00
Panu Matilainen a944143cf5 Clarify the macro entry struct a bit
- Macro name, opts and body must not be freed or otherwise written
  to after initialization, make them const pointers to clarify
  (and enforce) this. Arena is used to store whatever we need: it
  always contains the macro body, sometimes also name and/or opts,
  but these can be pointers to elsewhere too.
2013-04-04 11:24:01 +03:00
Alexey Tourbin 301d5450a1 Improve macro table performance
In the existing implementation, when a new macro is added, the whole
table has to be sorted again.  Hence the cost of adding n macros is
worse than O(n^2), due to arithmetic progression.

This change drops all qsort(3) stuff altogether, by carefully preserving
table in sorted order.  In findEntry routine, bsearch(3) is replaced
with customized binary search which tracks position for insertion.
In the addMacro routine, if a matching entry is not found, this
position is used for direct insertion, after the rest of the elements
are "shifted to the right" with memmove(3).  Likewise, in delMacro
routine, the elements are shifted back to the left when the last macro
definition is popped.  Technically, shifting half of the array with
memmove(3) is still O(n^2); however, modern CPUs process contiguous
memory in a very efficient manner, and glibc provides a fine-tuned
memmove(3) implementation.

Also, macro table entries are now allocated in a single chunk.

This change reduces rpm startup costs by factor of 6.  Also, this change
improves specfile parser performance by a factor of 2 (e.g. the parse
time of texlive.spec is reduced from 67s to 35s).

Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
2013-04-04 11:10:40 +03:00
Alexey Tourbin c22d5b1299 Stricter macro substitution syntax
This change introduces a separate routine to parse for valid macro
names.  Valid macro names are either regular 3+ character identifiers,
or special names: "S", "P", "0", "#", "*", "**", macro options such as
"-o" and "-o*", and macro arguments such as "1".  Other names are not
valid.  This fixes a number of bugs seen earlier due to sloppy name
parsing: "%_libdir*" and "%01" were not expanded (these are now expanded
to e.g. "/usr/lib64*" and "<name>1", as expected).  This also fixes
bugs in as-is substitution: "%!foo" was expanded to "%foo", and likewise
"%!!!" was expanded to "%" (and to "%<garbage>" at EOL).

Also, bad names in %name and %{name...} substitutions are now handled
differently.  In %name form, the name is parsed tentatively; a silent
fall-back to as-is substitution is provisioned when no valid name can
be obtain.  In %{name...} form, a failure to obtain a valid name is now
a syntax error.  Furthermore, only 3 variants are syntactically valid:
%{name} proper, %{name:...}, and %{name ...}.  This renders invalid
ambiguous macro substitutions such as the one found in FC18 lvm2.spec:

Requires: util-linux >= %{util-linux_version}
error: Invalid macro syntax: %{util-linux_version}
Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
2013-02-12 15:53:04 +02:00
Alexey Tourbin 2d647098c8 Fixed buffer problems in doShellEscape
When the output from a command is empty, nothing stops doShellEscape from
chopping newlines past the beginning of the buffer.  This problem was first
identified by Dmitry V. Levin in July 2009.

Also, there is an off-by-one error in replacing trailing '\n' with '\0'.
This problem, however, escaped the attention of Dmitry V. Levin in July 2009.

Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
2013-02-12 08:30:33 +02:00
Alexey Tourbin 0eb0da9af9 Warn when whitespace is missing before macro body
This will now issue a warning when macro definition is possibly
incorrect or ambigous, such as the one found in FC18 lvm2.spec:

%define util-linux_version 2.22.1
warning: Macro %util needs whitespace before body

Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
2013-02-09 14:36:30 +02:00
Thierry Vignaud 8192e580a0 display the actual unknown option instead of '?'
'?' is returned by getopt when option is unknown, making hard to
pinpoint the actual bogus option...

Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
2012-12-18 10:38:36 +02:00
Panu Matilainen 1fdcb341cd Enable built-in %dirname macro
- This might've clashed with %dir in spec %files at some point as
  indicated by the comments, but that should no longer be the case.
2012-11-01 12:12:51 +02:00
Christophe Fergeau 185596818f Add support for 7zip compressed tarballs
Teach %prep and %uncompress how to handle 7zip tarballs, with
the mingw toolchain landing in fedora, this may be useful when
crossbuilding Windows sources compressed using 7zip (CxImage is
one such project).
2012-05-22 13:48:01 +03:00
Panu Matilainen db782b4b62 Keep the macro buffer terminated on append
- This isn't strictly needed as we're terminating the buffers "just in
  case" all over the place but handling this centrally might allow
  some day eliminating the other fluff...
2012-05-10 11:49:24 +03:00
Panu Matilainen 6101cac6b3 Fix off-by-ones in mbAppend() and mbAppendStr()
- Oops, remember to reserve space for the trailing \0 when appending.
  mb->nb holds the number of actual characters left in the buffer,
  not the terminator. Fixes a regression introduced in rpm 4.9.x
  dynamic macro realloction work (RhBug:431009 reprise)
2012-05-10 11:28:39 +03:00
Panu Matilainen bd4fc30884 Clean up + clarify popMacro() a bit
- Actually protect against NULL mep (shouldn't happen but...)
- Remove bogus comment + add actually relevant comments
- Make the *mep reassignment more obvious by taking it out of
  the if where its easily missed
- Replace dead NULL-assignments with a trash-n-burn memset()
- Fixup indentation to match general rpm style
2011-05-31 11:38:38 +03:00
Panu Matilainen 6c391a88fa Use popMacro() when freeing the entire macro table
- We already have a function to free macro entries, use it to
  remove code duplication
2011-05-31 11:03:43 +03:00
Panu Matilainen 182e9f8a64 Eliminate bunch of obvious dead NULL-assignments around librpmio 2011-05-28 19:43:52 +03:00
Panu Matilainen ecfece7ec0 Add support for nested Lua macro expansion (RhBug:490740)
- Lift the printbuffer accounting out of rpmlua into a struct of
  its own (Funny thing, this looks a whole lot like the macro
  expansion buffer and Good Ole StringBuf Brothers ... Boys ... Mam.
  Unify them one of these days maybe)
- Replace the simplistic on/off printbuffer with a stack of buffers,
  fixup the lone caller to use the new internal push/pop API.
2011-05-24 20:41:38 +03:00
Panu Matilainen 0b36a1f18e Eliminate bogus ferror() checks
- fopen() returns NULL on errors, never an opened stream with error
  flag set. These are leftovers from past where rpmio fd was used
  instead of FILE and probably the checks were bogus even back then too.
2011-05-18 10:59:54 +03:00
Michael Schroeder f4c79584d0 Always copy macro source when expanding it
- A macro can undefine itself, and unless we grab a copy of it we'll
  end up accessing already freed memory. Fixes a regression from
  commit ebc4ceaaeb which assumed
  a copy is not always needed.

Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
2011-05-18 09:10:11 +03:00
Ville Skyttä e254f9b9d4 Add lrzip support.
Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
2011-04-29 08:49:05 +03:00
Ville Skyttä bcad08867d Add lzip support.
Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
2011-04-29 08:48:56 +03:00
Panu Matilainen 1817ac020e Plug memory leaks on macro definition error cases 2011-03-17 15:35:42 +02:00
Panu Matilainen 131d082aab Handle non-terminated substrings centrally in expandMacro()
- expandMacro() is big and ugly already, but handling the non-terminated
  substrings there once and for all avoids having to ping-pong recurse
  through couple of helpers to get there.
2010-09-27 17:15:15 +03:00
Panu Matilainen b1dbdf50af Dynamic expansion buffer reallocation for lua macros too 2010-09-27 16:32:15 +03:00
Panu Matilainen 800a1daa23 Whoops, fix typo/thinko from commit dc258bac41 2010-09-27 16:22:38 +03:00
Panu Matilainen 46f32d25ce Eliminate the internal in-place-expansion buffer limitation
- Return the dynamically allocated expansion buffer to callers from
  expandU(), except using a slightly less cryptic expandThis() name
  for it now. Also deal with non-terminated strings centrally in
  expandThis() instead of doing alloc + copy to terminate in every caller.
2010-09-27 16:10:36 +03:00
Panu Matilainen d8777387db Simplify / sanitize expandU() a bit
- Instead of saving and restoring the bits we'll mess with, just
  make a temporary expansion state struct with non-buffer state
  copied from "parent".
2010-09-27 14:55:12 +03:00
Panu Matilainen 49f99b8656 Dynamically reallocate macro expansion buffer (ticket #45)
- Eliminate the underlying limitation of macro expansion limit by
  growing the buffer as needed when adding characters to it. This
  doesn't fix the entire issue yet however: expandU() and expandMacros()
  are still limited to caller-specified buffer size, and output
  from lua-macros isn't dynamically resized. One step at a time...
2010-09-27 14:55:12 +03:00
Panu Matilainen 61a3fc4523 Let expandMacro() allocate its own buffer
- More pre-requisites for dynamic buffer resizing, callers have no
  clue how much expandMacro() is going to need.
2010-09-27 14:55:12 +03:00
Panu Matilainen dc258bac41 Track current expansion point via distance to buffer start
- Supposedly no functional changes, just paving way for dynamic
  buffer resizing which is impossible when accessing pointers directly.
2010-09-27 14:55:12 +03:00
Panu Matilainen 568ba0d242 Push the macro buffer size limit down by one level
- Turn expandMacros() into a wrapper around internal doExpandMacros()
  which returns the expanded string in a newly allocated buffer, use
  the internal version for rpmExpand() too.
2010-09-27 14:55:12 +03:00
Panu Matilainen a931202422 Eliminate unused spec field from macro expansion state struct 2010-09-27 14:55:11 +03:00
Panu Matilainen 8082eb48db Avoid calloc() in macro findEntry()
- It's just the macro name we're grabbing here, that ought to fit
  on stack comfortably enough.
2010-09-27 14:55:11 +03:00
Panu Matilainen ebc4ceaaeb Avoid unnecessary round-trip through expandT() where possible
- In the cases where expandT() was called with strlen(source) we can
  now just bypass it and call expandMacro() directly, avoiding an
  unnecessary string copy.
2010-09-27 14:55:11 +03:00
Panu Matilainen ccba5236a0 Use function arguments to pass, duh, arguments into functions. Duh.
- No functional changes, just eliminating mindless save-and-restore
  ping-pong through macro expansion state struct.
2010-09-27 14:55:11 +03:00
Jindrich Novy 7f220202f2 Redirect macro stack backtrace printing to debug log. 2010-07-13 13:35:22 +02:00
Jindrich Novy 60be56e317 Fix cryptic macro evaluation error message (RhBug:613010) 2010-07-13 13:34:05 +02:00
Panu Matilainen c16cde7a4d Revert "Always free locally defined macros when they go out of scope"
- the "fix" breaks seemingly legitimate use in fedora font package
  macros, possibly some funky interaction with %{lua: } macros or something
- this reverts commit f895acd285.
2010-01-07 12:04:53 +02:00
Panu Matilainen f895acd285 Always free locally defined macros when they go out of scope
- Prior to this, local defines in constructs like %{!?foo: %define foo bar}
  would remain defined until a parametrized macro gets called, causing
  obscure and confusing errors in specs such as RhBug:551971 and countless
  others. Use of %global is now always required for such constructs.
2010-01-04 12:45:18 +02:00
Panu Matilainen 2c5ea0b188 Remove -98 vintage ifdef'ed dead code 2010-01-04 10:52:22 +02:00
Panu Matilainen 2553ca2fe3 Move getopt() portability fiddles out of system.h
- only macro.c needs, no point polluting includes everywhere
2009-12-23 14:02:19 +02:00
Ville Skyttä 1892984f02 Decompress lzma files with xz. 2009-10-06 10:47:36 +03:00
Panu Matilainen f634253b32 Eliminate unused variable 2009-09-07 17:50:43 +03:00
Panu Matilainen f4b39519d3 Replace equal/not equal uses of str[n]cmp() with rstreq[n] in io code 2009-08-31 11:15:16 +03:00
Panu Matilainen 3d63df6220 Handle XZ in %uncompress macro
- should've been in commit 8078d0ba24
2009-03-27 14:11:43 +02:00
Panu Matilainen 2d2bda29c1 Add built-in %getconfdir macro for getting rpmConfigDir() value
- define %_rpmconfigdir via %getconfdir in the main macro config, this
  avoids it getting lost on macro reloads as happens when building
  several packages at once
2009-03-27 14:01:47 +02:00
Panu Matilainen 947ad7acb3 Add getenv macro, use instead of $(echo $HOME) for defauls
- inspired by http://lists.rpm.org/pipermail/rpm-maint/2009-January/002332.html
2009-01-16 09:09:40 +02:00
Panu Matilainen da83a562e1 Add function to retrieve rpm config base directory
- one-shot to determine configuration base directory path from
  RPM_CONFIGDIR environement or build-time default
- rpmfileutil is a bit strange place, this would really belong to librpm
  but Lua initialization needs the path so...
2008-11-22 18:06:31 +02:00
Panu Matilainen 49c515179a Permit tab as whitespace between macro arguments (rhbz#467567)
- another dumb regression from macro handling cleanups
2008-10-19 01:53:03 +03:00
Panu Matilainen 9f85707825 Macro argument handling regression regression (rhbz#461180)
- commit f366011c42 is too simplistic and
  broke other stuff in turn...
2008-09-11 16:56:50 +03:00
Panu Matilainen 1d9b4d00d7 Make rpmExpand() smarter wrt memory allocations
- precalculate unexpanded size and allocate enough for that plus MACROBUFSIZ
  for expansion
- typical allocation is way smaller than what gets allocated "just in case",
  calculate expanded size and realloc to actual size to avoid wasting
  memory
2008-07-16 10:52:33 +03:00
Panu Matilainen 889f9f5b41 Eliminate another unnecessary temp variable
- do all the copying inside a loop
2008-07-16 10:33:57 +03:00
Panu Matilainen 1b19676027 Eliminate unnecessary variable
- leftover from static buffer times..
2008-07-16 10:24:05 +03:00
Panu Matilainen f366011c42 Macro argument handling regression (rhbz#455333)
- expandMacro() wants the next \0 character to be returned, which might
  or might not be the same as lastc passed to grabArgs()
- use memcpy() instead of memmove() for the copy, the areas can't overlap
2008-07-16 09:40:10 +03:00
Panu Matilainen af9518dce2 Use correct type for strlen() 2008-06-10 10:02:52 +03:00
Panu Matilainen 485d195488 Plug memleak in macro doFoo() 2008-04-29 17:39:02 +03:00
Panu Matilainen 9ae911eefe Further simplify rpmInitMacros()
- Use argvSplit() for splitting the macro path to components instead of
  manual pointer-parsery.
- If URL's are to be supported or accepted at all (previous code attempted
  to skip them), ':' is a very poor delimiter character.
2008-04-19 16:42:56 +03:00
Panu Matilainen e336146a9e Use argvFree() to free, well, an argv, instead of manual freeing 2008-04-19 16:04:11 +03:00
Panu Matilainen 601b4d0e6a Avoid entirely silly rpmio FD_t use
- rdcl() converts to FILE internally anyway, might as well use fopen()
  to begin with...
2008-04-18 16:14:15 +03:00
Panu Matilainen 38a975b157 Remove unnecessary xstrdup() of dynamically allocated buffer
- leftover from when target buffer was static...
2008-04-18 15:39:07 +03:00
Panu Matilainen 94e64fec18 Cosmetics: group operations more logically... 2008-04-18 15:15:05 +03:00
Panu Matilainen 2d8a27ed16 Simplify macro option + argument processing
- eliminate ninja-voodoo pointer-hackery parsing, replace argvSplit() +
  argvJoin() and friends
- eliminate static sized buffers from the processing
2008-04-18 14:59:03 +03:00
Panu Matilainen 02939c15fa More accurate filtering of .rpmnew etc when reading macros
- strstr() can give false positives, use rpmFileHasSuffix() instead
2008-04-17 17:33:30 +03:00
Panu Matilainen 3c1126b2a5 Missing decompression arguments for bzip in uncompress macro 2008-04-04 15:52:52 +03:00
Panu Matilainen 7ddab3ab8d Dead code removal (macro stuff) 2008-03-31 10:44:23 +03:00
Panu Matilainen c908277be2 Minimal namespacing for locale-insensitive x*() string functions 2008-03-18 09:10:13 +02:00
Panu Matilainen 05129f83dc Make bunch of macro internal variables static 2008-02-26 22:24:28 +02:00
Panu Matilainen cc69dd20b8 Allocate macro buffers dynamically
- avoids some entirely needless overflow cases
- somewhat similarly to rpm5.org but use heap instead of stack
2008-02-01 13:00:40 +02:00
Panu Matilainen 56405587f4 Allocate various macro buffers from heap not stack 2008-02-01 12:11:03 +02:00
Panu Matilainen 54ab76019e Define + use MACROBUFSIZ for macro expansion buffer size
- double the buffer size while at it...
2008-02-01 11:36:13 +02:00
Panu Matilainen d3d60512bd Macro shell expansion fixes (rhbz#431009) 2008-02-01 11:09:13 +02:00
Panu Matilainen eb5dc35c19 Include spring-cleaning
- put some consistency into include ordering
- everything (apart from bits missed ;) is now ordered like this
  1. "system.h"
  2. other system includes
  3. rpm public headers
  4. rpm private headers
  5. "debug.h"
2008-01-30 17:05:29 +02:00
Panu Matilainen 95b03441cc Rename isCompressed() -> rpmFileIsCompressed() for namespacing 2008-01-27 16:39:40 +02:00
Panu Matilainen eec03952be Make uncompress macros match reality (_ vs __ prefix) 2008-01-23 07:56:51 +02:00
Panu Matilainen 8e00018c88 Fix some const issues in macros 2007-12-19 12:49:52 +02:00
Panu Matilainen 78ad426f04 Remove bogus const from rpmGlob argv return type
- it's malloced and must be freed by caller
2007-12-18 10:17:18 +02:00
Panu Matilainen c41f61369e Unconstify macro contents, fully contained within macro module 2007-12-16 21:28:37 +02:00
Panu Matilainen f5a712db5b Remove bogus consts.. 2007-12-14 15:50:17 +02:00
Panu Matilainen 9ce13e09ef Switch to <rpm/foo.h> style for public headers
- adjust include paths accordingly
2007-12-08 14:02:32 +02:00
Panu Matilainen 89527785bb Move logging related macros rpmlog, adjust includes 2007-11-30 23:08:38 +02:00
Ralf Corsépius 8ef0ca213f Expand private include file names to be relative to $(top_srcdir) 2007-11-23 11:41:29 +01:00
Panu Matilainen 7b2a9b3c24 Move rpmGlob() from macros to rpmfileutil.h
- macros is all just macros now
- internal copy of popt stuff needs to go...
2007-11-23 10:27:29 +02:00
Panu Matilainen 2c6905dd9f Move rpmGetPath and rpmGenPath to rpmfileutil.h 2007-11-23 10:20:19 +02:00
Panu Matilainen 178e32a0ba Move rpmCleanPath from macros to rpmfileutil.h 2007-11-23 10:10:19 +02:00
Panu Matilainen ff77bfa2a1 Move file compression stuff from rpmmacro.h to rpmfileutil.h 2007-11-23 10:05:49 +02:00
Panu Matilainen 4ad615daf5 Move the xstr*() string variants from rpmio.h to rpmstring.h 2007-11-23 08:32:08 +02:00
Ralf Corsépius abeea80a38 Use #include <x.h> syntax to include public headers. 2007-11-23 06:46:19 +01:00
Panu Matilainen 17ef8a3082 Lose rpmerr.h for good 2007-11-19 16:32:15 +02:00